chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,48 @@
|
||||
🆕 Stochastic Training of GNNs with GraphBolt
|
||||
=============================================
|
||||
|
||||
GraphBolt is a data loading framework for GNN with high flexibility and
|
||||
scalability. It is built on top of DGL and PyTorch.
|
||||
|
||||
This tutorial introduces how to enable stochastic training of GNNs with
|
||||
GraphBolt.
|
||||
|
||||
Overview
|
||||
^^^^^^^
|
||||
|
||||
.. image:: ../_static/graphbolt_overview.jpg
|
||||
:width: 700
|
||||
:alt: Graphbolt Overview
|
||||
|
||||
GraphBolt integrates seamlessly with the PyTorch `datapipe <https://pytorch.org/data/beta/torchdata.datapipes.iter.html>`_, relying on the unified "MiniBatch" data structure to connect processing stages. It streamlines data loading and preprocessing for GNN training, validation, and testing.
|
||||
By default, GraphBolt provides a collection of built-in datasets and exceptionally efficient implementations of datapipes for common scenarios, which can be summarized as follows:
|
||||
|
||||
1. **Item Sampler:** Randomly selects a subset (nodes, edges, graphs) from the entire training set as an initial mini-batch for downstream computation.
|
||||
|
||||
2. **Negative Sampler:** Specially designed for link prediction tasks, it generates non-existing edges as negative examples for training.
|
||||
|
||||
3. **Subgraph Sampler:** Generates subgraphs based on the input nodes/edges for computation.
|
||||
|
||||
4. **Feature Fetcher:** Fetches related node/edge features from the dataset for the given input.
|
||||
|
||||
By exposing the entire data loading process as a pipeline, GraphBolt provides significant flexibility and customization opportunities. Users can easily substitute any stage with their own implementations. Additionally, users can benefit from the optimized scheduling strategy for datapipes, even with customized stages.
|
||||
|
||||
In summary, GraphBolt offers the following benefits:
|
||||
|
||||
1. A flexible, pipelined framework for GNN data loading and preprocessing.
|
||||
|
||||
2. Highly efficient canonical implementations.
|
||||
|
||||
3. Efficient scheduling.
|
||||
|
||||
Scenarios
|
||||
^^^^^^^
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
neighbor_sampling_overview.nblink
|
||||
node_classification.nblink
|
||||
link_prediction.nblink
|
||||
multigpu_node_classification.nblink
|
||||
ondisk-dataset.rst
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/link_prediction.ipynb"
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/multigpu_node_classification.ipynb"
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/neighbor_sampling_overview.ipynb"
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/node_classification.ipynb"
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
.. _stochastic_training-ondisk-dataset-specification:
|
||||
|
||||
YAML specification
|
||||
==================
|
||||
|
||||
This document describes the YAML specification of ``metadata.yaml`` file for
|
||||
``OnDiskDataset``. ``metadata.yaml`` file is used to specify the dataset
|
||||
information, including the graph structure, feature data and tasks.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
dataset_name: <string>
|
||||
graph:
|
||||
nodes:
|
||||
- type: <string>
|
||||
num: <int>
|
||||
- type: <string>
|
||||
num: <int>
|
||||
edges:
|
||||
- type: <string>
|
||||
format: <string>
|
||||
path: <string>
|
||||
- type: <string>
|
||||
format: <string>
|
||||
path: <string>
|
||||
feature_data:
|
||||
- domain: node
|
||||
type: <string>
|
||||
name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- domain: node
|
||||
type: <string>
|
||||
name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- domain: edge
|
||||
type: <string>
|
||||
name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- domain: edge
|
||||
type: <string>
|
||||
name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
tasks:
|
||||
- name: <string>
|
||||
num_classes: <int>
|
||||
train_set:
|
||||
- type: <string>
|
||||
data:
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
validation_set:
|
||||
- type: <string>
|
||||
data:
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
test_set:
|
||||
- type: <string>
|
||||
data:
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
- name: <string>
|
||||
format: <string>
|
||||
in_memory: <bool>
|
||||
path: <string>
|
||||
|
||||
``dataset_name``
|
||||
---------------
|
||||
|
||||
The ``dataset_name`` field is used to specify the name of the dataset. It is
|
||||
user-defined.
|
||||
|
||||
``graph``
|
||||
---------
|
||||
|
||||
The ``graph`` field is used to specify the graph structure. It has two fields:
|
||||
``nodes`` and ``edges``.
|
||||
|
||||
- ``nodes``: ``list``
|
||||
|
||||
The ``nodes`` field is used to specify the number of nodes for each node type.
|
||||
It is a list of ``node`` objects. Each ``node`` object has two fields: ``type``
|
||||
and ``num``.
|
||||
- ``type``: ``string``, optional
|
||||
|
||||
The ``type`` field is used to specify the node type. It is ``null`` for
|
||||
homogeneous graphs. For heterogeneous graphs, it is the node type.
|
||||
- ``num``: ``int``
|
||||
|
||||
The ``num`` field is used to specify the number of nodes for the node type.
|
||||
It is mandatory for both homogeneous graphs and heterogeneous graphs.
|
||||
|
||||
- ``edges``: ``list``
|
||||
|
||||
The ``edges`` field is used to specify the edges. It is a list of ``edge``
|
||||
objects. Each ``edge`` object has three fields: ``type``, ``format`` and
|
||||
``path``.
|
||||
- ``type``: ``string``, optional
|
||||
|
||||
The ``type`` field is used to specify the edge type. It is ``null`` for
|
||||
homogeneous graphs. For heterogeneous graphs, it is the edge type.
|
||||
- ``format``: ``string``
|
||||
|
||||
The ``format`` field is used to specify the format of the edge data. It
|
||||
can be ``csv`` or ``numpy``. If it is ``csv``, no ``index`` and ``header``
|
||||
fields are needed. If it is ``numpy``, the array requires to be in shape
|
||||
of ``(2, num_edges)``. ``numpy`` format is recommended for large graphs.
|
||||
- ``path``: ``string``
|
||||
|
||||
The ``path`` field is used to specify the path of the edge data. It is
|
||||
relative to the directory of ``metadata.yaml`` file.
|
||||
|
||||
|
||||
``feature_data``
|
||||
----------------
|
||||
|
||||
The ``feature_data`` field is used to specify the feature data. It is a list of
|
||||
``feature`` objects. Each ``feature`` object has five canonical fields: ``domain``,
|
||||
``type``, ``name``, ``format`` and ``path``. Any other fields will be passed to
|
||||
the ``Feature.metadata`` object.
|
||||
|
||||
- ``domain``: ``string``
|
||||
|
||||
The ``domain`` field is used to specify the domain of the feature data. It can
|
||||
be either ``node`` or ``edge``.
|
||||
- ``type``: ``string``, optional
|
||||
|
||||
The ``type`` field is used to specify the type of the feature data. It is
|
||||
``null`` for homogeneous graphs. For heterogeneous graphs, it is the node or
|
||||
edge type.
|
||||
- ``name``: ``string``
|
||||
|
||||
The ``name`` field is used to specify the name of the feature data. It is
|
||||
user-defined.
|
||||
- ``format``: ``string``
|
||||
|
||||
The ``format`` field is used to specify the format of the feature data. It can
|
||||
be either ``numpy`` or ``torch``.
|
||||
- ``in_memory``: ``bool``, optional
|
||||
|
||||
The ``in_memory`` field is used to specify whether the feature data is loaded
|
||||
into memory. It can be either ``true`` or ``false``. Default is ``true``.
|
||||
- ``path``: ``string``
|
||||
|
||||
The ``path`` field is used to specify the path of the feature data. It is
|
||||
relative to the directory of ``metadata.yaml`` file.
|
||||
|
||||
|
||||
``tasks``
|
||||
---------
|
||||
|
||||
The ``tasks`` field is used to specify the tasks. It is a list of ``task``
|
||||
objects. Each ``task`` object has at least three fields: ``train_set``,
|
||||
``validation_set``, ``test_set``. And you are free to add other fields
|
||||
such as ``num_classes`` and all these fields will be passed to the
|
||||
``Task.metadata`` object.
|
||||
|
||||
- ``name``: ``string``, optional
|
||||
|
||||
The ``name`` field is used to specify the name of the task. It is user-defined.
|
||||
- ``num_classes``: ``int``, optional
|
||||
|
||||
The ``num_classes`` field is used to specify the number of classes of the task.
|
||||
- ``train_set``: ``list``
|
||||
|
||||
The ``train_set`` field is used to specify the training set. It is a list of
|
||||
``set`` objects. Each ``set`` object has two fields: ``type`` and ``data``.
|
||||
- ``type``: ``string``, optional
|
||||
|
||||
The ``type`` field is used to specify the node/edge type of the set. It is
|
||||
``null`` for homogeneous graphs. For heterogeneous graphs, it is the node
|
||||
or edge type.
|
||||
- ``data``: ``list``
|
||||
|
||||
The ``data`` field is used to load ``train_set``. It is a list of ``data``
|
||||
objects. Each ``data`` object has four fields: ``name``, ``format``,
|
||||
``in_memory`` and ``path``.
|
||||
|
||||
- ``name``: ``string``
|
||||
|
||||
The ``name`` field is used to specify the name of the data. It is mandatory
|
||||
and used to specify the data fields of ``MiniBatch`` for sampling. It can
|
||||
be either ``seeds``, ``labels`` or ``indexes``. If any other name is used,
|
||||
it will be added into the ``MiniBatch`` data fields.
|
||||
- ``format``: ``string``
|
||||
|
||||
The ``format`` field is used to specify the format of the data. It can be
|
||||
either ``numpy`` or ``torch``.
|
||||
- ``in_memory``: ``bool``, optional
|
||||
|
||||
The ``in_memory`` field is used to specify whether the data is loaded into
|
||||
memory. It can be either ``true`` or ``false``. Default is ``true``.
|
||||
- ``path``: ``string``
|
||||
|
||||
The ``path`` field is used to specify the path of the data. It is relative
|
||||
to the directory of ``metadata.yaml`` file.
|
||||
- ``validation_set``: ``list``
|
||||
- ``test_set``: ``list``
|
||||
|
||||
The ``validation_set`` and ``test_set`` fields are used to specify the
|
||||
validation set and test set respectively. They are similar to the
|
||||
``train_set`` field.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
.. _stochastic_training-ondisk-dataset:
|
||||
|
||||
Composing OnDiskDataset from raw data
|
||||
=====================================
|
||||
|
||||
This tutorial shows how to compose :class:`~dgl.graphbolt.OnDiskDataset` from
|
||||
raw data. A full specification of ``metadata.yaml`` is also provided.
|
||||
|
||||
**GraphBolt** provides the ``OnDiskDataset`` class to help user organize plain
|
||||
data of graph strucutre, feature data and tasks. ``OnDiskDataset`` is also
|
||||
designed to efficiently handle large graphs and features that do not fit into
|
||||
memory by storing them on disk.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:glob:
|
||||
|
||||
ondisk_dataset_homograph.nblink
|
||||
ondisk_dataset_heterograph.nblink
|
||||
ondisk-dataset-specification.rst
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/ondisk_dataset_heterograph.ipynb"
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"path": "../../../notebooks/stochastic_training/ondisk_dataset_homograph.ipynb"
|
||||
}
|
||||
Reference in New Issue
Block a user