chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,276 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"accelerator": "GPU",
|
||||
"gpuClass": "standard"
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Building a Graph Convolutional Network Using Sparse Matrices\n",
|
||||
"\n",
|
||||
"This tutorial illustrates step-by-step how to write and train a Graph Convolutional Network ([Kipf et al. (2017)](https://arxiv.org/abs/1609.02907)) using DGL's sparse matrix APIs.\n",
|
||||
"\n",
|
||||
"[](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/gcn.ipynb) [](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/gcn.ipynb)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "_iqWrPwxtZr6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Install required packages.\n",
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"os.environ['TORCH'] = torch.__version__\n",
|
||||
"os.environ['DGLBACKEND'] = \"pytorch\"\n",
|
||||
"\n",
|
||||
"# Uncomment below to install required packages. If the CUDA version is not 11.8,\n",
|
||||
"# check the https://www.dgl.ai/pages/start.html to find the supported CUDA\n",
|
||||
"# version and corresponding command to install DGL.\n",
|
||||
"#!pip install dgl -f https://data.dgl.ai/wheels/cu118/repo.html > /dev/null\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import dgl\n",
|
||||
" installed = True\n",
|
||||
"except ImportError:\n",
|
||||
" installed = False\n",
|
||||
"print(\"DGL installed!\" if installed else \"DGL not found!\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FTqB360eRvya"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Graph Convolutional Layer\n",
|
||||
"\n",
|
||||
"Mathematically, the graph convolutional layer is defined as:\n",
|
||||
"\n",
|
||||
"$$f(X^{(l)}, A) = \\sigma(\\bar{D}^{-\\frac{1}{2}}\\bar{A}\\bar{D}^{-\\frac{1}{2}}X^{(l)}W^{(l)})$$\n",
|
||||
"\n",
|
||||
"with $\\bar{A} = A + I$, where $A$ denotes the adjacency matrix and $I$ denotes the identity matrix, $\\bar{D}$ refers to the diagonal node degree matrix of $\\bar{A}$ and $W^{(l)}$ denotes a trainable weight matrix. $\\sigma$ refers to a non-linear activation (e.g. relu).\n",
|
||||
"\n",
|
||||
"The code below shows how to implement it using the `dgl.sparse` package. The core operations are:\n",
|
||||
"\n",
|
||||
"* `dgl.sparse.identity` creates the identity matrix $I$.\n",
|
||||
"* The augmented adjacency matrix $\\bar{A}$ is then computed by adding the identity matrix to the adjacency matrix $A$.\n",
|
||||
"* `A_hat.sum(0)` aggregates the augmented adjacency matrix $\\bar{A}$ along the first dimension which gives the degree vector of the augmented graph. The diagonal degree matrix $\\bar{D}$ is then created by `dgl.sparse.diag`.\n",
|
||||
"* Compute $\\bar{D}^{-\\frac{1}{2}}$.\n",
|
||||
"* `D_hat_invsqrt @ A_hat @ D_hat_invsqrt` computes the convolution matrix which is then multiplied by the linearly transformed node features."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "r3qB1atg_ld0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"\n",
|
||||
"import dgl.sparse as dglsp\n",
|
||||
"\n",
|
||||
"class GCNLayer(nn.Module):\n",
|
||||
" def __init__(self, in_size, out_size):\n",
|
||||
" super(GCNLayer, self).__init__()\n",
|
||||
" self.W = nn.Linear(in_size, out_size)\n",
|
||||
"\n",
|
||||
" def forward(self, A, X):\n",
|
||||
" ########################################################################\n",
|
||||
" # (HIGHLIGHT) Compute the symmetrically normalized adjacency matrix with\n",
|
||||
" # Sparse Matrix API\n",
|
||||
" ########################################################################\n",
|
||||
" I = dglsp.identity(A.shape)\n",
|
||||
" A_hat = A + I\n",
|
||||
" D_hat = dglsp.diag(A_hat.sum(0))\n",
|
||||
" D_hat_invsqrt = D_hat ** -0.5\n",
|
||||
" return D_hat_invsqrt @ A_hat @ D_hat_invsqrt @ self.W(X)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Y4I4EhHQ_kKb"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"A Graph Convolutional Network is then defined by stacking this layer."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bvP7O2IwV_c7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Create a GCN with the GCN layer.\n",
|
||||
"class GCN(nn.Module):\n",
|
||||
" def __init__(self, in_size, out_size, hidden_size):\n",
|
||||
" super(GCN, self).__init__()\n",
|
||||
" self.conv1 = GCNLayer(in_size, hidden_size)\n",
|
||||
" self.conv2 = GCNLayer(hidden_size, out_size)\n",
|
||||
"\n",
|
||||
" def forward(self, A, X):\n",
|
||||
" X = self.conv1(A, X)\n",
|
||||
" X = F.relu(X)\n",
|
||||
" return self.conv2(A, X)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "BHX3vRjDWJTO"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Training the GCN\n",
|
||||
"\n",
|
||||
"We then train the GCN model on the Cora dataset for node classification. Note that since the model expects an adjacency matrix as the first argument, we first construct the adjacency matrix from the graph using the `dgl.sparse.from_coo` API which returns a DGL `SparseMatrix` object."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "2Qw7fTdGNnEp"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def evaluate(g, pred):\n",
|
||||
" label = g.ndata[\"label\"]\n",
|
||||
" val_mask = g.ndata[\"val_mask\"]\n",
|
||||
" test_mask = g.ndata[\"test_mask\"]\n",
|
||||
"\n",
|
||||
" # Compute accuracy on validation/test set.\n",
|
||||
" val_acc = (pred[val_mask] == label[val_mask]).float().mean()\n",
|
||||
" test_acc = (pred[test_mask] == label[test_mask]).float().mean()\n",
|
||||
" return val_acc, test_acc\n",
|
||||
"\n",
|
||||
"def train(model, g):\n",
|
||||
" features = g.ndata[\"feat\"]\n",
|
||||
" label = g.ndata[\"label\"]\n",
|
||||
" train_mask = g.ndata[\"train_mask\"]\n",
|
||||
" optimizer = torch.optim.Adam(model.parameters(), lr=1e-2, weight_decay=5e-4)\n",
|
||||
" loss_fcn = nn.CrossEntropyLoss()\n",
|
||||
"\n",
|
||||
" # Preprocess to get the adjacency matrix of the graph.\n",
|
||||
" indices = torch.stack(g.edges())\n",
|
||||
" N = g.num_nodes()\n",
|
||||
" A = dglsp.spmatrix(indices, shape=(N, N))\n",
|
||||
"\n",
|
||||
" for epoch in range(100):\n",
|
||||
" model.train()\n",
|
||||
"\n",
|
||||
" # Forward.\n",
|
||||
" logits = model(A, features)\n",
|
||||
"\n",
|
||||
" # Compute loss with nodes in the training set.\n",
|
||||
" loss = loss_fcn(logits[train_mask], label[train_mask])\n",
|
||||
"\n",
|
||||
" # Backward.\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
" # Compute prediction.\n",
|
||||
" pred = logits.argmax(dim=1)\n",
|
||||
"\n",
|
||||
" # Evaluate the prediction.\n",
|
||||
" val_acc, test_acc = evaluate(g, pred)\n",
|
||||
" if epoch % 5 == 0:\n",
|
||||
" print(\n",
|
||||
" f\"In epoch {epoch}, loss: {loss:.3f}, val acc: {val_acc:.3f}\"\n",
|
||||
" f\", test acc: {test_acc:.3f}\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Load graph from the existing dataset.\n",
|
||||
"dataset = dgl.data.CoraGraphDataset()\n",
|
||||
"g = dataset[0]\n",
|
||||
"\n",
|
||||
"# Create model.\n",
|
||||
"feature = g.ndata['feat']\n",
|
||||
"in_size = feature.shape[1]\n",
|
||||
"out_size = dataset.num_classes\n",
|
||||
"gcn_model = GCN(in_size, out_size, 16)\n",
|
||||
"\n",
|
||||
"# Kick off training.\n",
|
||||
"train(gcn_model, g)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "5Sp1B1_QHgC2",
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"outputId": "552e2c22-44f4-4495-c7f9-a57f13484270"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Downloading /root/.dgl/cora_v2.zip from https://data.dgl.ai/dataset/cora_v2.zip...\n",
|
||||
"Extracting file to /root/.dgl/cora_v2\n",
|
||||
"Finished data loading and preprocessing.\n",
|
||||
" NumNodes: 2708\n",
|
||||
" NumEdges: 10556\n",
|
||||
" NumFeats: 1433\n",
|
||||
" NumClasses: 7\n",
|
||||
" NumTrainingSamples: 140\n",
|
||||
" NumValidationSamples: 500\n",
|
||||
" NumTestSamples: 1000\n",
|
||||
"Done saving data into cached files.\n",
|
||||
"In epoch 0, loss: 1.954, val acc: 0.114, test acc: 0.103\n",
|
||||
"In epoch 5, loss: 1.921, val acc: 0.158, test acc: 0.147\n",
|
||||
"In epoch 10, loss: 1.878, val acc: 0.288, test acc: 0.283\n",
|
||||
"In epoch 15, loss: 1.822, val acc: 0.344, test acc: 0.353\n",
|
||||
"In epoch 20, loss: 1.751, val acc: 0.388, test acc: 0.389\n",
|
||||
"In epoch 25, loss: 1.663, val acc: 0.406, test acc: 0.410\n",
|
||||
"In epoch 30, loss: 1.562, val acc: 0.472, test acc: 0.481\n",
|
||||
"In epoch 35, loss: 1.450, val acc: 0.558, test acc: 0.573\n",
|
||||
"In epoch 40, loss: 1.333, val acc: 0.636, test acc: 0.641\n",
|
||||
"In epoch 45, loss: 1.216, val acc: 0.684, test acc: 0.683\n",
|
||||
"In epoch 50, loss: 1.102, val acc: 0.726, test acc: 0.713\n",
|
||||
"In epoch 55, loss: 0.996, val acc: 0.740, test acc: 0.740\n",
|
||||
"In epoch 60, loss: 0.899, val acc: 0.754, test acc: 0.760\n",
|
||||
"In epoch 65, loss: 0.813, val acc: 0.762, test acc: 0.771\n",
|
||||
"In epoch 70, loss: 0.737, val acc: 0.768, test acc: 0.781\n",
|
||||
"In epoch 75, loss: 0.671, val acc: 0.776, test acc: 0.786\n",
|
||||
"In epoch 80, loss: 0.614, val acc: 0.784, test acc: 0.790\n",
|
||||
"In epoch 85, loss: 0.566, val acc: 0.780, test acc: 0.788\n",
|
||||
"In epoch 90, loss: 0.524, val acc: 0.780, test acc: 0.791\n",
|
||||
"In epoch 95, loss: 0.489, val acc: 0.772, test acc: 0.795\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"*Check out the full example script* [here](https://github.com/dmlc/dgl/blob/master/examples/sparse/gcn.py)."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "yQnJZvE9ZduM"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,410 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "Jv-tHPvR-JKa"
|
||||
},
|
||||
"source": [
|
||||
"# Graph Transformer in a Nutshell\n",
|
||||
"\n",
|
||||
"The **Transformer** [(Vaswani et al. 2017)](https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html) has been proven an effective learning architecture in natural language processing and computer vision.\n",
|
||||
"Recently, researchers turns to explore the application of transformer in graph learning. They have achieved inital success on many practical tasks, e.g., graph property prediction.\n",
|
||||
"[Dwivedi et al. (2020)](https://arxiv.org/abs/2012.09699) firstly generalize the transformer neural architecture to graph-structured data. Here, we present how to build such a graph transformer with DGL's sparse matrix APIs.\n",
|
||||
"\n",
|
||||
"[](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/graph_transformer.ipynb) [](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/graph_transformer.ipynb)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Install required packages.\n",
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"os.environ['TORCH'] = torch.__version__\n",
|
||||
"os.environ['DGLBACKEND'] = \"pytorch\"\n",
|
||||
"\n",
|
||||
"# Uncomment below to install required packages. If the CUDA version is not 11.8,\n",
|
||||
"# check the https://www.dgl.ai/pages/start.html to find the supported CUDA\n",
|
||||
"# version and corresponding command to install DGL.\n",
|
||||
"#!pip install dgl -f https://data.dgl.ai/wheels/cu118/repo.html > /dev/null\n",
|
||||
"#!pip install ogb >/dev/null\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import dgl\n",
|
||||
" installed = True\n",
|
||||
"except ImportError:\n",
|
||||
" installed = False\n",
|
||||
"print(\"DGL installed!\" if installed else \"Failed to install DGL!\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "8wIJZQqODy-7"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "nOpFdtLI-JKb"
|
||||
},
|
||||
"source": [
|
||||
"## Sparse Multi-head Attention\n",
|
||||
"\n",
|
||||
"Recall the all-pairs scaled-dot-product attention mechanism in vanillar Transformer:\n",
|
||||
"\n",
|
||||
"$$\\text{Attn}=\\text{softmax}(\\dfrac{QK^T} {\\sqrt{d}})V,$$\n",
|
||||
"\n",
|
||||
"The graph transformer (GT) model employs a Sparse Multi-head Attention block:\n",
|
||||
"\n",
|
||||
"$$\\text{SparseAttn}(Q, K, V, A) = \\text{softmax}(\\frac{(QK^T) \\circ A}{\\sqrt{d}})V,$$\n",
|
||||
"\n",
|
||||
"where $Q, K, V ∈\\mathbb{R}^{N\\times d}$ are query feature, key feature, and value feature, respectively. $A\\in[0,1]^{N\\times N}$ is the adjacency matrix of the input graph. $(QK^T)\\circ A$ means that the multiplication of query matrix and key matrix is followed by a Hadamard product (or element-wise multiplication) with the sparse adjacency matrix as illustrated in the figure below:\n",
|
||||
"\n",
|
||||
"<img src=\"https://drive.google.com/uc?id=1OgMAewLR3Z1vz5y4J8aPRSeaU3g8iQfX\" width=\"500\">\n",
|
||||
"\n",
|
||||
"Essentially, only the attention scores between connected nodes are computed according to the sparsity of $A$. This operation is also called *Sampled Dense Dense Matrix Multiplication (SDDMM)*.\n",
|
||||
"\n",
|
||||
"Enjoying the [batched SDDMM API](https://docs.dgl.ai/en/latest/generated/dgl.sparse.bsddmm.html) in DGL, we can parallel the computation on multiple attention heads (different representation subspaces).\n",
|
||||
"\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "dh7zc5v0-JKb"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import dgl\n",
|
||||
"import dgl.nn as dglnn\n",
|
||||
"import dgl.sparse as dglsp\n",
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import torch.optim as optim\n",
|
||||
"\n",
|
||||
"from dgl.data import AsGraphPredDataset\n",
|
||||
"from dgl.dataloading import GraphDataLoader\n",
|
||||
"from ogb.graphproppred import collate_dgl, DglGraphPropPredDataset, Evaluator\n",
|
||||
"from ogb.graphproppred.mol_encoder import AtomEncoder\n",
|
||||
"from tqdm import tqdm\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class SparseMHA(nn.Module):\n",
|
||||
" \"\"\"Sparse Multi-head Attention Module\"\"\"\n",
|
||||
"\n",
|
||||
" def __init__(self, hidden_size=80, num_heads=8):\n",
|
||||
" super().__init__()\n",
|
||||
" self.hidden_size = hidden_size\n",
|
||||
" self.num_heads = num_heads\n",
|
||||
" self.head_dim = hidden_size // num_heads\n",
|
||||
" self.scaling = self.head_dim**-0.5\n",
|
||||
"\n",
|
||||
" self.q_proj = nn.Linear(hidden_size, hidden_size)\n",
|
||||
" self.k_proj = nn.Linear(hidden_size, hidden_size)\n",
|
||||
" self.v_proj = nn.Linear(hidden_size, hidden_size)\n",
|
||||
" self.out_proj = nn.Linear(hidden_size, hidden_size)\n",
|
||||
"\n",
|
||||
" def forward(self, A, h):\n",
|
||||
" N = len(h)\n",
|
||||
" # [N, dh, nh]\n",
|
||||
" q = self.q_proj(h).reshape(N, self.head_dim, self.num_heads)\n",
|
||||
" q *= self.scaling\n",
|
||||
" # [N, dh, nh]\n",
|
||||
" k = self.k_proj(h).reshape(N, self.head_dim, self.num_heads)\n",
|
||||
" # [N, dh, nh]\n",
|
||||
" v = self.v_proj(h).reshape(N, self.head_dim, self.num_heads)\n",
|
||||
"\n",
|
||||
" ######################################################################\n",
|
||||
" # (HIGHLIGHT) Compute the multi-head attention with Sparse Matrix API\n",
|
||||
" ######################################################################\n",
|
||||
" attn = dglsp.bsddmm(A, q, k.transpose(1, 0)) # (sparse) [N, N, nh]\n",
|
||||
" # Sparse softmax by default applies on the last sparse dimension.\n",
|
||||
" attn = attn.softmax() # (sparse) [N, N, nh]\n",
|
||||
" out = dglsp.bspmm(attn, v) # [N, dh, nh]\n",
|
||||
"\n",
|
||||
" return self.out_proj(out.reshape(N, -1))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "3_Fm6Lrx-JKc"
|
||||
},
|
||||
"source": [
|
||||
"## Graph Transformer Layer\n",
|
||||
"\n",
|
||||
"The GT layer is composed of Multi-head Attention, Batch Norm, and Feed-forward Network, connected by residual links as in vanilla transformer.\n",
|
||||
"\n",
|
||||
"<img src=\"https://drive.google.com/uc?id=1cm-Ijw7bUQIOkoTKn5MQ3m4-66JqCsMz\" width=\"300\">"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "M6h7JVWT-JKd"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class GTLayer(nn.Module):\n",
|
||||
" \"\"\"Graph Transformer Layer\"\"\"\n",
|
||||
"\n",
|
||||
" def __init__(self, hidden_size=80, num_heads=8):\n",
|
||||
" super().__init__()\n",
|
||||
" self.MHA = SparseMHA(hidden_size=hidden_size, num_heads=num_heads)\n",
|
||||
" self.batchnorm1 = nn.BatchNorm1d(hidden_size)\n",
|
||||
" self.batchnorm2 = nn.BatchNorm1d(hidden_size)\n",
|
||||
" self.FFN1 = nn.Linear(hidden_size, hidden_size * 2)\n",
|
||||
" self.FFN2 = nn.Linear(hidden_size * 2, hidden_size)\n",
|
||||
"\n",
|
||||
" def forward(self, A, h):\n",
|
||||
" h1 = h\n",
|
||||
" h = self.MHA(A, h)\n",
|
||||
" h = self.batchnorm1(h + h1)\n",
|
||||
"\n",
|
||||
" h2 = h\n",
|
||||
" h = self.FFN2(F.relu(self.FFN1(h)))\n",
|
||||
" h = h2 + h\n",
|
||||
"\n",
|
||||
" return self.batchnorm2(h)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "t40DhVjI-JKd"
|
||||
},
|
||||
"source": [
|
||||
"## Graph Transformer Model\n",
|
||||
"\n",
|
||||
"The GT model is constructed by stacking GT layers. The input positional encoding of vanilla transformer is replaced with Laplacian positional encoding [(Dwivedi et al. 2020)](https://arxiv.org/abs/2003.00982). For the graph-level prediction task, an extra pooler is stacked on top of GT layers to aggregate node feature of the same graph."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "UrjvEBrF-JKe"
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class GTModel(nn.Module):\n",
|
||||
" def __init__(\n",
|
||||
" self,\n",
|
||||
" out_size,\n",
|
||||
" hidden_size=80,\n",
|
||||
" pos_enc_size=2,\n",
|
||||
" num_layers=8,\n",
|
||||
" num_heads=8,\n",
|
||||
" ):\n",
|
||||
" super().__init__()\n",
|
||||
" self.atom_encoder = AtomEncoder(hidden_size)\n",
|
||||
" self.pos_linear = nn.Linear(pos_enc_size, hidden_size)\n",
|
||||
" self.layers = nn.ModuleList(\n",
|
||||
" [GTLayer(hidden_size, num_heads) for _ in range(num_layers)]\n",
|
||||
" )\n",
|
||||
" self.pooler = dglnn.SumPooling()\n",
|
||||
" self.predictor = nn.Sequential(\n",
|
||||
" nn.Linear(hidden_size, hidden_size // 2),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_size // 2, hidden_size // 4),\n",
|
||||
" nn.ReLU(),\n",
|
||||
" nn.Linear(hidden_size // 4, out_size),\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" def forward(self, g, X, pos_enc):\n",
|
||||
" indices = torch.stack(g.edges())\n",
|
||||
" N = g.num_nodes()\n",
|
||||
" A = dglsp.spmatrix(indices, shape=(N, N))\n",
|
||||
" h = self.atom_encoder(X) + self.pos_linear(pos_enc)\n",
|
||||
" for layer in self.layers:\n",
|
||||
" h = layer(A, h)\n",
|
||||
" h = self.pooler(g, h)\n",
|
||||
"\n",
|
||||
" return self.predictor(h)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {
|
||||
"id": "RdrPU18I-JKe"
|
||||
},
|
||||
"source": [
|
||||
"## Training\n",
|
||||
"\n",
|
||||
"We train the GT model on [ogbg-molhiv](https://ogb.stanford.edu/docs/graphprop/#ogbg-mol) benchmark. The Laplacian positional encoding of each graph is pre-computed (with the API [here](https://docs.dgl.ai/en/latest/generated/dgl.laplacian_pe.html)) as part of the input to the model.\n",
|
||||
"\n",
|
||||
"*Note that we down-sample the dataset to make this demo runs faster. See the* [*example script*](https://github.com/dmlc/dgl/blob/master/examples/sparse/graph_transformer.py) *for the performance on the full dataset.*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"id": "V41i0w-9-JKe",
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"outputId": "15343d1a-a32d-4677-d053-d9da96910f43"
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stderr",
|
||||
"text": [
|
||||
"Computing Laplacian PE: 1%| | 25/4000 [00:00<00:16, 244.77it/s]/usr/local/lib/python3.8/dist-packages/dgl/backend/pytorch/tensor.py:52: UserWarning: Casting complex values to real discards the imaginary part (Triggered internally at ../aten/src/ATen/native/Copy.cpp:250.)\n",
|
||||
" return th.as_tensor(data, dtype=dtype)\n",
|
||||
"Computing Laplacian PE: 100%|██████████| 4000/4000 [00:13<00:00, 296.04it/s]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Epoch: 000, Loss: 0.2486, Val: 0.3082, Test: 0.3068\n",
|
||||
"Epoch: 001, Loss: 0.1695, Val: 0.4684, Test: 0.4572\n",
|
||||
"Epoch: 002, Loss: 0.1428, Val: 0.5887, Test: 0.4721\n",
|
||||
"Epoch: 003, Loss: 0.1237, Val: 0.6375, Test: 0.5010\n",
|
||||
"Epoch: 004, Loss: 0.1127, Val: 0.6628, Test: 0.4854\n",
|
||||
"Epoch: 005, Loss: 0.1047, Val: 0.6811, Test: 0.4983\n",
|
||||
"Epoch: 006, Loss: 0.0949, Val: 0.6751, Test: 0.5409\n",
|
||||
"Epoch: 007, Loss: 0.0901, Val: 0.6340, Test: 0.5357\n",
|
||||
"Epoch: 008, Loss: 0.0811, Val: 0.6717, Test: 0.5543\n",
|
||||
"Epoch: 009, Loss: 0.0643, Val: 0.7861, Test: 0.5628\n",
|
||||
"Epoch: 010, Loss: 0.0489, Val: 0.7319, Test: 0.5341\n",
|
||||
"Epoch: 011, Loss: 0.0340, Val: 0.7884, Test: 0.5299\n",
|
||||
"Epoch: 012, Loss: 0.0285, Val: 0.5887, Test: 0.4293\n",
|
||||
"Epoch: 013, Loss: 0.0361, Val: 0.5514, Test: 0.3419\n",
|
||||
"Epoch: 014, Loss: 0.0451, Val: 0.6795, Test: 0.4964\n",
|
||||
"Epoch: 015, Loss: 0.0429, Val: 0.7405, Test: 0.5527\n",
|
||||
"Epoch: 016, Loss: 0.0331, Val: 0.7859, Test: 0.4994\n",
|
||||
"Epoch: 017, Loss: 0.0177, Val: 0.6544, Test: 0.4457\n",
|
||||
"Epoch: 018, Loss: 0.0201, Val: 0.8250, Test: 0.6073\n",
|
||||
"Epoch: 019, Loss: 0.0093, Val: 0.7356, Test: 0.5561\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"@torch.no_grad()\n",
|
||||
"def evaluate(model, dataloader, evaluator, device):\n",
|
||||
" model.eval()\n",
|
||||
" y_true = []\n",
|
||||
" y_pred = []\n",
|
||||
" for batched_g, labels in dataloader:\n",
|
||||
" batched_g, labels = batched_g.to(device), labels.to(device)\n",
|
||||
" y_hat = model(batched_g, batched_g.ndata[\"feat\"], batched_g.ndata[\"PE\"])\n",
|
||||
" y_true.append(labels.view(y_hat.shape).detach().cpu())\n",
|
||||
" y_pred.append(y_hat.detach().cpu())\n",
|
||||
" y_true = torch.cat(y_true, dim=0).numpy()\n",
|
||||
" y_pred = torch.cat(y_pred, dim=0).numpy()\n",
|
||||
" input_dict = {\"y_true\": y_true, \"y_pred\": y_pred}\n",
|
||||
" return evaluator.eval(input_dict)[\"rocauc\"]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def train(model, dataset, evaluator, device):\n",
|
||||
" train_dataloader = GraphDataLoader(\n",
|
||||
" dataset[dataset.train_idx],\n",
|
||||
" batch_size=256,\n",
|
||||
" shuffle=True,\n",
|
||||
" collate_fn=collate_dgl,\n",
|
||||
" )\n",
|
||||
" valid_dataloader = GraphDataLoader(\n",
|
||||
" dataset[dataset.val_idx], batch_size=256, collate_fn=collate_dgl\n",
|
||||
" )\n",
|
||||
" test_dataloader = GraphDataLoader(\n",
|
||||
" dataset[dataset.test_idx], batch_size=256, collate_fn=collate_dgl\n",
|
||||
" )\n",
|
||||
" optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
|
||||
" num_epochs = 20\n",
|
||||
" scheduler = optim.lr_scheduler.StepLR(\n",
|
||||
" optimizer, step_size=num_epochs, gamma=0.5\n",
|
||||
" )\n",
|
||||
" loss_fcn = nn.BCEWithLogitsLoss()\n",
|
||||
"\n",
|
||||
" for epoch in range(num_epochs):\n",
|
||||
" model.train()\n",
|
||||
" total_loss = 0.0\n",
|
||||
" for batched_g, labels in train_dataloader:\n",
|
||||
" batched_g, labels = batched_g.to(device), labels.to(device)\n",
|
||||
" logits = model(\n",
|
||||
" batched_g, batched_g.ndata[\"feat\"], batched_g.ndata[\"PE\"]\n",
|
||||
" )\n",
|
||||
" loss = loss_fcn(logits, labels.float())\n",
|
||||
" total_loss += loss.item()\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
" scheduler.step()\n",
|
||||
" avg_loss = total_loss / len(train_dataloader)\n",
|
||||
" val_metric = evaluate(model, valid_dataloader, evaluator, device)\n",
|
||||
" test_metric = evaluate(model, test_dataloader, evaluator, device)\n",
|
||||
" print(\n",
|
||||
" f\"Epoch: {epoch:03d}, Loss: {avg_loss:.4f}, \"\n",
|
||||
" f\"Val: {val_metric:.4f}, Test: {test_metric:.4f}\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Training device.\n",
|
||||
"dev = torch.device(\"cpu\")\n",
|
||||
"# Uncomment the code below to train on GPU. Be sure to install DGL with CUDA support.\n",
|
||||
"#dev = torch.device(\"cuda:0\")\n",
|
||||
"\n",
|
||||
"# Load dataset.\n",
|
||||
"pos_enc_size = 8\n",
|
||||
"dataset = AsGraphPredDataset(\n",
|
||||
" DglGraphPropPredDataset(\"ogbg-molhiv\", \"./data/OGB\")\n",
|
||||
")\n",
|
||||
"evaluator = Evaluator(\"ogbg-molhiv\")\n",
|
||||
"\n",
|
||||
"# Down sample the dataset to make the tutorial run faster.\n",
|
||||
"import random\n",
|
||||
"random.seed(42)\n",
|
||||
"train_size = len(dataset.train_idx)\n",
|
||||
"val_size = len(dataset.val_idx)\n",
|
||||
"test_size = len(dataset.test_idx)\n",
|
||||
"dataset.train_idx = dataset.train_idx[\n",
|
||||
" torch.LongTensor(random.sample(range(train_size), 2000))\n",
|
||||
"]\n",
|
||||
"dataset.val_idx = dataset.val_idx[\n",
|
||||
" torch.LongTensor(random.sample(range(val_size), 1000))\n",
|
||||
"]\n",
|
||||
"dataset.test_idx = dataset.test_idx[\n",
|
||||
" torch.LongTensor(random.sample(range(test_size), 1000))\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Laplacian positional encoding.\n",
|
||||
"indices = torch.cat([dataset.train_idx, dataset.val_idx, dataset.test_idx])\n",
|
||||
"for idx in tqdm(indices, desc=\"Computing Laplacian PE\"):\n",
|
||||
" g, _ = dataset[idx]\n",
|
||||
" g.ndata[\"PE\"] = dgl.laplacian_pe(g, k=pos_enc_size, padding=True)\n",
|
||||
"\n",
|
||||
"# Create model.\n",
|
||||
"out_size = dataset.num_tasks\n",
|
||||
"model = GTModel(out_size=out_size, pos_enc_size=pos_enc_size).to(dev)\n",
|
||||
"\n",
|
||||
"# Kick off training.\n",
|
||||
"train(model, dataset, evaluator, dev)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"orig_nbformat": 4,
|
||||
"colab": {
|
||||
"provenance": []
|
||||
},
|
||||
"gpuClass": "standard",
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"accelerator": "GPU"
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@@ -0,0 +1,401 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"provenance": [],
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"gpuClass": "standard",
|
||||
"accelerator": "GPU"
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Hypergraph Neural Networks\n",
|
||||
"\n",
|
||||
"This tutorial illustrates what is hypergraph and how to build a Hypergraph Neural Network using DGL's sparse matrix APIs.\n",
|
||||
"\n",
|
||||
"[](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/hgnn.ipynb) [](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/hgnn.ipynb)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "eiDu3XgReCt4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"# Install required packages.\n",
|
||||
"import os\n",
|
||||
"import torch\n",
|
||||
"os.environ['TORCH'] = torch.__version__\n",
|
||||
"os.environ['DGLBACKEND'] = \"pytorch\"\n",
|
||||
"\n",
|
||||
"# Uncomment below to install required packages. If the CUDA version is not 11.8,\n",
|
||||
"# check the https://www.dgl.ai/pages/start.html to find the supported CUDA\n",
|
||||
"# version and corresponding command to install DGL.\n",
|
||||
"#!pip install dgl -f https://data.dgl.ai/wheels/cu118/repo.html > /dev/null\n",
|
||||
"#!pip install torchmetrics > /dev/null\n",
|
||||
"\n",
|
||||
"try:\n",
|
||||
" import dgl\n",
|
||||
" installed = True\n",
|
||||
"except ImportError:\n",
|
||||
" installed = False\n",
|
||||
"print(\"DGL installed!\" if installed else \"Failed to install DGL!\")"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "__2tKqL0eaB0"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Hypergraphs\n",
|
||||
"\n",
|
||||
"A [hypergraph](https://en.wikipedia.org/wiki/Hypergraph) consists of *nodes* and *hyperedges*. Contrary to edges in graphs, a *hyperedge* can connect arbitrary number of nodes. For instance, the following figure shows a hypergraph with 11 nodes and 5 hyperedges drawn in different colors.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Hypergraphs are particularly useful when the relationships between data points within the dataset is not binary. For instance, more than two products can be co-purchased together in an e-commerce system, so the relationship of co-purchase is $n$-ary rather than binary, and therefore it is better described as a hypergraph rather than a normal graph.\n",
|
||||
"\n",
|
||||
"A hypergraph is usually characterized by its *incidence matrix* $H$, whose rows represent nodes and columns represent hyperedges. An entry $H_{ij}$ is 1 if hyperedge $j$ includes node $i$, or 0 otherwise. For example, the hypergraph in the figure above can be characterized by a $11 \\times 5$ matrix as follows:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"H = \\begin{bmatrix}\n",
|
||||
"1 & 0 & 0 & 0 & 0 \\\\\n",
|
||||
"1 & 0 & 0 & 0 & 0 \\\\\n",
|
||||
"1 & 1 & 0 & 1 & 1 \\\\\n",
|
||||
"0 & 0 & 1 & 0 & 0 \\\\\n",
|
||||
"0 & 1 & 0 & 0 & 0 \\\\\n",
|
||||
"1 & 0 & 1 & 1 & 1 \\\\\n",
|
||||
"0 & 0 & 1 & 0 & 0 \\\\\n",
|
||||
"0 & 1 & 0 & 1 & 0 \\\\\n",
|
||||
"0 & 1 & 0 & 1 & 0 \\\\\n",
|
||||
"0 & 0 & 1 & 0 & 1 \\\\\n",
|
||||
"0 & 0 & 0 & 0 & 1 \\\\\n",
|
||||
"\\end{bmatrix}\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"One can construct the hypergraph incidence matrix by specifying two tensors `nodes` and `hyperedges`, where the node ID `nodes[i]` belongs to the hyperedge ID `hyperedges[i]` for all `i`. In the case above, the incidence matrix can be constructed below.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "unL_mAj-TqC6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import dgl.sparse as dglsp\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"H = dglsp.spmatrix(\n",
|
||||
" torch.LongTensor([[0, 1, 2, 2, 2, 2, 3, 4, 5, 5, 5, 5, 6, 7, 7, 8, 8, 9, 9, 10],\n",
|
||||
" [0, 0, 0, 1, 3, 4, 2, 1, 0, 2, 3, 4, 2, 1, 3, 1, 3, 2, 4, 4]])\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"print(H.to_dense())"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "I_cExvtIJD1F",
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"outputId": "a1a576f6-1559-479c-9f3e-93e41a56833d"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"tensor([[1., 0., 0., 0., 0.],\n",
|
||||
" [1., 0., 0., 0., 0.],\n",
|
||||
" [1., 1., 0., 1., 1.],\n",
|
||||
" [0., 0., 1., 0., 0.],\n",
|
||||
" [0., 1., 0., 0., 0.],\n",
|
||||
" [1., 0., 1., 1., 1.],\n",
|
||||
" [0., 0., 1., 0., 0.],\n",
|
||||
" [0., 1., 0., 1., 0.],\n",
|
||||
" [0., 1., 0., 1., 0.],\n",
|
||||
" [0., 0., 1., 0., 1.],\n",
|
||||
" [0., 0., 0., 0., 1.]])\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"The degree of a node in a hypergraph is defined as the number of hyperedges including the node. Similarly, the degree of a hyperedge in a hypergraph is defined as the number of nodes included by the hyperedge. In the example above, the hyperedge degrees can be computed by the sum of row vectors (i.e. all 4), while the node degree can be computed by the sum of column vectors."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "p-shCPQPHvBB"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"node_degrees = H.sum(1)\n",
|
||||
"print(\"Node degrees\", node_degrees)\n",
|
||||
"\n",
|
||||
"hyperedge_degrees = H.sum(0)\n",
|
||||
"print(\"Hyperedge degrees\", hyperedge_degrees)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "wjKm9gkTOnU9",
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"outputId": "ffe2c441-8c2c-48a7-cef2-4ef6e96548ec"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Node degrees tensor([1., 1., 4., 1., 1., 4., 1., 2., 2., 2., 1.])\n",
|
||||
"Hyperedge degrees tensor([4., 4., 4., 4., 4.])\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"\n",
|
||||
"## Hypergraph Neural Network (HGNN) Layer\n",
|
||||
"\n",
|
||||
"The [HGNN layer](https://arxiv.org/pdf/1809.09401.pdf) is defined as:\n",
|
||||
"\n",
|
||||
"$$f(X^{(l)}, H; W^{(l)}) = \\sigma(L X^{(l)} W^{(l)})$$$$L = D_v^{-1/2} H B D_e^{-1} H^\\top D_v^{-1/2}$$\n",
|
||||
"\n",
|
||||
"where\n",
|
||||
"\n",
|
||||
"* $H \\in \\mathbb{R}^{N \\times M}$ is the incidence matrix of hypergraph with $N$ nodes and $M$ hyperedges.\n",
|
||||
"* $D_v \\in \\mathbb{R}^{N \\times N}$ is a diagonal matrix representing node degrees, whose $i$-th diagonal element is $\\sum_{j=1}^M H_{ij}$.\n",
|
||||
"* $D_e \\in \\mathbb{R}^{M \\times M}$ is a diagonal matrix representing hyperedge degrees, whose $j$-th diagonal element is $\\sum_{i=1}^N H_{ij}$.\n",
|
||||
"* $B \\in \\mathbb{R}^{M \\times M}$ is a diagonal matrix representing the hyperedge weights, whose $j$-th diagonal element is the weight of $j$-th hyperedge. In our example, $B$ is an identity matrix.\n",
|
||||
"\n",
|
||||
"The following code builds a two-layer HGNN."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "7kxrINkVHrAi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"import dgl.sparse as dglsp\n",
|
||||
"import torch\n",
|
||||
"import torch.nn as nn\n",
|
||||
"import torch.nn.functional as F\n",
|
||||
"import tqdm\n",
|
||||
"from dgl.data import CoraGraphDataset\n",
|
||||
"from torchmetrics.functional import accuracy\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class HGNN(nn.Module):\n",
|
||||
" def __init__(self, H, in_size, out_size, hidden_dims=16):\n",
|
||||
" super().__init__()\n",
|
||||
"\n",
|
||||
" self.W1 = nn.Linear(in_size, hidden_dims)\n",
|
||||
" self.W2 = nn.Linear(hidden_dims, out_size)\n",
|
||||
" self.dropout = nn.Dropout(0.5)\n",
|
||||
"\n",
|
||||
" ###########################################################\n",
|
||||
" # (HIGHLIGHT) Compute the Laplacian with Sparse Matrix API\n",
|
||||
" ###########################################################\n",
|
||||
" # Compute node degree.\n",
|
||||
" d_V = H.sum(1)\n",
|
||||
" # Compute edge degree.\n",
|
||||
" d_E = H.sum(0)\n",
|
||||
" # Compute the inverse of the square root of the diagonal D_v.\n",
|
||||
" D_v_invsqrt = dglsp.diag(d_V**-0.5)\n",
|
||||
" # Compute the inverse of the diagonal D_e.\n",
|
||||
" D_e_inv = dglsp.diag(d_E**-1)\n",
|
||||
" # In our example, B is an identity matrix.\n",
|
||||
" n_edges = d_E.shape[0]\n",
|
||||
" B = dglsp.identity((n_edges, n_edges))\n",
|
||||
" # Compute Laplacian from the equation above.\n",
|
||||
" self.L = D_v_invsqrt @ H @ B @ D_e_inv @ H.T @ D_v_invsqrt\n",
|
||||
"\n",
|
||||
" def forward(self, X):\n",
|
||||
" X = self.L @ self.W1(self.dropout(X))\n",
|
||||
" X = F.relu(X)\n",
|
||||
" X = self.L @ self.W2(self.dropout(X))\n",
|
||||
" return X"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "58WnPtPvT2mx"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Loading Data\n",
|
||||
"\n",
|
||||
"We use Cora citation network in our example. But instead of using the original \"cite\" relationship between papers, we consider the \"co-cite\" relationship between papers. We build a hypergraph from the original citation network where for each paper we construct a hyperedge that includes all the other papers it cited, as well as the paper itself.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Note that a hypergraph constructed this way has an incidence matrix exactly identical to the adjacency matrix of the original graph (plus an identity matrix for self-loops). This is because each hyperedge has a one-to-one correspondence to each paper. So we can directly take the graph's adjacency matrix and add an identity matrix to it, and we use it as the hypergraph's incidence matrix."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "bPrOHVaGwUD0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def load_data():\n",
|
||||
" dataset = CoraGraphDataset()\n",
|
||||
"\n",
|
||||
" graph = dataset[0]\n",
|
||||
" indices = torch.stack(graph.edges())\n",
|
||||
" H = dglsp.spmatrix(indices)\n",
|
||||
" H = H + dglsp.identity(H.shape)\n",
|
||||
"\n",
|
||||
" X = graph.ndata[\"feat\"]\n",
|
||||
" Y = graph.ndata[\"label\"]\n",
|
||||
" train_mask = graph.ndata[\"train_mask\"]\n",
|
||||
" val_mask = graph.ndata[\"val_mask\"]\n",
|
||||
" test_mask = graph.ndata[\"test_mask\"]\n",
|
||||
" return H, X, Y, dataset.num_classes, train_mask, val_mask, test_mask"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "qI0j1J9pwTFg"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Training and Evaluation\n",
|
||||
"\n",
|
||||
"Now we can write the training and evaluation functions as follows."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "--rq1-r7wMST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"source": [
|
||||
"def train(model, optimizer, X, Y, train_mask):\n",
|
||||
" model.train()\n",
|
||||
" Y_hat = model(X)\n",
|
||||
" loss = F.cross_entropy(Y_hat[train_mask], Y[train_mask])\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
" loss.backward()\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"def evaluate(model, X, Y, val_mask, test_mask, num_classes):\n",
|
||||
" model.eval()\n",
|
||||
" Y_hat = model(X)\n",
|
||||
" val_acc = accuracy(\n",
|
||||
" Y_hat[val_mask], Y[val_mask], task=\"multiclass\", num_classes=num_classes\n",
|
||||
" )\n",
|
||||
" test_acc = accuracy(\n",
|
||||
" Y_hat[test_mask],\n",
|
||||
" Y[test_mask],\n",
|
||||
" task=\"multiclass\",\n",
|
||||
" num_classes=num_classes,\n",
|
||||
" )\n",
|
||||
" return val_acc, test_acc\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"H, X, Y, num_classes, train_mask, val_mask, test_mask = load_data()\n",
|
||||
"model = HGNN(H, X.shape[1], num_classes)\n",
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
|
||||
"\n",
|
||||
"with tqdm.trange(500) as tq:\n",
|
||||
" for epoch in tq:\n",
|
||||
" train(model, optimizer, X, Y, train_mask)\n",
|
||||
" val_acc, test_acc = evaluate(\n",
|
||||
" model, X, Y, val_mask, test_mask, num_classes\n",
|
||||
" )\n",
|
||||
" tq.set_postfix(\n",
|
||||
" {\n",
|
||||
" \"Val acc\": f\"{val_acc:.5f}\",\n",
|
||||
" \"Test acc\": f\"{test_acc:.5f}\",\n",
|
||||
" },\n",
|
||||
" refresh=False,\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
"print(f\"Test acc: {test_acc:.3f}\")"
|
||||
],
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"id": "IfEc6JRXwHPt",
|
||||
"outputId": "0172578a-6a1b-49eb-adcb-77ee1a949186"
|
||||
},
|
||||
"execution_count": null,
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Downloading /root/.dgl/cora_v2.zip from https://data.dgl.ai/dataset/cora_v2.zip...\n",
|
||||
"Extracting file to /root/.dgl/cora_v2\n",
|
||||
"Finished data loading and preprocessing.\n",
|
||||
" NumNodes: 2708\n",
|
||||
" NumEdges: 10556\n",
|
||||
" NumFeats: 1433\n",
|
||||
" NumClasses: 7\n",
|
||||
" NumTrainingSamples: 140\n",
|
||||
" NumValidationSamples: 500\n",
|
||||
" NumTestSamples: 1000\n",
|
||||
"Done saving data into cached files.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stderr",
|
||||
"text": [
|
||||
"100%|██████████| 500/500 [00:57<00:00, 8.70it/s, Val acc=0.77800, Test acc=0.78100]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stdout",
|
||||
"text": [
|
||||
"Test acc: 0.781\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stderr",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"For the complete example of HGNN, please refer to [here](https://github.com/dmlc/dgl/blob/master/examples/sparse/hgnn.py)."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "59pCzjpBOyEW"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user