wehub-resource-sync
1310 lines
40 KiB
Plaintext
1310 lines
40 KiB
Plaintext
{
|
||
"nbformat": 4,
|
||
"nbformat_minor": 0,
|
||
"metadata": {
|
||
"colab": {
|
||
"provenance": [],
|
||
"private_outputs": true,
|
||
"toc_visible": true,
|
||
"gpuType": "T4"
|
||
},
|
||
"kernelspec": {
|
||
"name": "python3",
|
||
"display_name": "Python 3"
|
||
},
|
||
"language_info": {
|
||
"name": "python"
|
||
}
|
||
},
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"# Quickstart\n",
|
||
"\n",
|
||
"The tutorial provides a quick walkthrough of the classes and operators provided by the `dgl.sparse` package.\n",
|
||
"\n",
|
||
"[](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb) [](https://github.com/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb)"
|
||
],
|
||
"metadata": {
|
||
"id": "E0DAKDMuWz7I"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"# Install the required packages.\n",
|
||
"\n",
|
||
"import os\n",
|
||
"# Uncomment following commands to download Pytorch and DGL\n",
|
||
"# !pip install torch==2.0.0+cpu torchvision==0.15.1+cpu torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cpu > /dev/null\n",
|
||
"# !pip install dgl==1.1.0 -f https://data.dgl.ai/wheels/repo.html > /dev/null\n",
|
||
"import torch\n",
|
||
"os.environ['TORCH'] = torch.__version__\n",
|
||
"os.environ['DGLBACKEND'] = \"pytorch\"\n",
|
||
"\n",
|
||
"\n",
|
||
"try:\n",
|
||
" import dgl.sparse as dglsp\n",
|
||
" installed = True\n",
|
||
"except ImportError:\n",
|
||
" installed = False\n",
|
||
"print(\"DGL installed!\" if installed else \"DGL not found!\")"
|
||
],
|
||
"metadata": {
|
||
"id": "19UZd7wyWzpT"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Sparse Matrix\n",
|
||
"\n",
|
||
"The core abstraction of DGL's sparse package is the `SparseMatrix` class. Compared with other sparse matrix libraries (such as `scipy.sparse` and `torch.sparse`), DGL's `SparseMatrix` is specialized for the deep learning workloads on structure data (e.g., Graph Neural Networks), with the following features:\n",
|
||
"\n",
|
||
"* **Auto sparse format.** Don't bother choosing between different sparse formats. There is only one `SparseMatrix` and it will select the best format for the operation to be performed.\n",
|
||
"* **Non-zero elements can be scalar or vector.** Easy for modeling relations (e.g., edges) by vector representation.\n",
|
||
"* **Fully PyTorch compatible.** The package is built upon PyTorch and is natively compatible with other tools in the PyTorch ecosystem.\n"
|
||
],
|
||
"metadata": {
|
||
"id": "GsWoAGC4RpHw"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### Creating a DGL Sparse Matrix\n",
|
||
"\n",
|
||
"The simplest way to create a sparse matrix is using the `spmatrix` API by providing the indices of the non-zero elements. The indices are stored in a tensor of shape `(2, nnz)`, where the `i`-th non-zero element is stored at position `(indices[0][i], indices[1][i])`. The code below creates a 3x3 sparse matrix.\n"
|
||
],
|
||
"metadata": {
|
||
"id": "_q4HYodcWenB"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": null,
|
||
"metadata": {
|
||
"id": "h-ryVEs1PuIP"
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import torch\n",
|
||
"import dgl.sparse as dglsp\n",
|
||
"\n",
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"A = dglsp.spmatrix(i) # 1.0 is default value for nnz elements.\n",
|
||
"\n",
|
||
"print(A)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.to_dense())"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"If not specified, the shape is inferred automatically from the indices but you can specify it explicitly too."
|
||
],
|
||
"metadata": {
|
||
"id": "W1JJg-eZ7K3t"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 0, 1],\n",
|
||
" [0, 2, 0]])\n",
|
||
"\n",
|
||
"A1 = dglsp.spmatrix(i)\n",
|
||
"print(f\"Implicit Shape: {A1.shape}\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"A2 = dglsp.spmatrix(i, shape=(3, 3))\n",
|
||
"print(f\"Explicit Shape: {A2.shape}\")\n",
|
||
"print(A2.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "80NNSQfd7L5V"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"Both scalar values and vector values can be set for nnz elements in Sparse Matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "zdNgUf0ShfCe"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"# The length of the value should match the nnz elements represented by the\n",
|
||
"# sparse matrix format.\n",
|
||
"scalar_val = torch.tensor([1., 2., 3.])\n",
|
||
"vector_val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
|
||
"\n",
|
||
"print(\"-----Scalar Values-----\")\n",
|
||
"A = dglsp.spmatrix(i, scalar_val)\n",
|
||
"print(A)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"print(\"-----Vector Values-----\")\n",
|
||
"A = dglsp.spmatrix(i, vector_val)\n",
|
||
"print(A)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "buE9ZkKvhp1f"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"*Duplicated indices*"
|
||
],
|
||
"metadata": {
|
||
"id": "7ufTCDAVsrmP"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 0, 0, 1],\n",
|
||
" [0, 2, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"print(A)\n",
|
||
"print(f\"Whether A contains duplicate indices: {A.has_duplicate()}\")\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"B = A.coalesce()\n",
|
||
"print(B)\n",
|
||
"print(f\"Whether B contains duplicate indices: {B.has_duplicate()}\")"
|
||
],
|
||
"metadata": {
|
||
"id": "ilSAlFLOs0o8"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**val_like**\n",
|
||
"\n",
|
||
"You can create a new sparse matrix by retaining the non-zero indices of a given sparse matrix but with different non-zero values."
|
||
],
|
||
"metadata": {
|
||
"id": "ZJ09qM5NaxuI"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"\n",
|
||
"new_val = torch.tensor([4., 5., 6.])\n",
|
||
"B = dglsp.val_like(A, new_val)\n",
|
||
"print(B)"
|
||
],
|
||
"metadata": {
|
||
"id": "UB3lKJVBbsUD"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**Create a sparse matrix from various sparse formats**\n",
|
||
"\n",
|
||
"* `from_coo()`: Create a sparse matrix from [COO](https://en.wikipedia.org/wiki/Sparse_matrix#Coordinate_list_(COO)) format.\n",
|
||
"* `from_csr()`: Create a sparse matrix from [CSR](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_row_(CSR,_CRS_or_Yale_format)) format.\n",
|
||
"* `from_csc()`: Create a sparse matrix from [CSC](https://en.wikipedia.org/wiki/Sparse_matrix#Compressed_sparse_column_(CSC_or_CCS)) format."
|
||
],
|
||
"metadata": {
|
||
"id": "nWjBSFDBXDPJ"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"row = torch.tensor([0, 1, 2, 2, 2])\n",
|
||
"col = torch.tensor([1, 2, 0, 1, 2])\n",
|
||
"\n",
|
||
"print(\"-----Create from COO format-----\")\n",
|
||
"A = dglsp.from_coo(row, col)\n",
|
||
"print(A)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"indptr = torch.tensor([0, 1, 2, 5])\n",
|
||
"indices = torch.tensor([1, 2, 0, 1, 2])\n",
|
||
"\n",
|
||
"print(\"-----Create from CSR format-----\")\n",
|
||
"A = dglsp.from_csr(indptr, indices)\n",
|
||
"print(A)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"print(\"-----Create from CSC format-----\")\n",
|
||
"B = dglsp.from_csc(indptr, indices)\n",
|
||
"print(B)\n",
|
||
"print(\"\")\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(B.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "3puXyMFsvdlj"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### Attributes and methods of a DGL Sparse Matrix"
|
||
],
|
||
"metadata": {
|
||
"id": "nd4hJ9ysd4St"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"\n",
|
||
"print(f\"Shape of sparse matrix: {A.shape}\")\n",
|
||
"print(f\"The number of nonzero elements of sparse matrix: {A.nnz}\")\n",
|
||
"print(f\"Datatype of sparse matrix: {A.dtype}\")\n",
|
||
"print(f\"Device sparse matrix is stored on: {A.device}\")\n",
|
||
"print(f\"Get the values of the nonzero elements: {A.val}\")\n",
|
||
"print(f\"Get the row indices of the nonzero elements: {A.row}\")\n",
|
||
"print(f\"Get the column indices of the nonzero elements: {A.col}\")\n",
|
||
"print(f\"Get the coordinate (COO) representation: {A.coo()}\")\n",
|
||
"print(f\"Get the compressed sparse row (CSR) representation: {A.csr()}\")\n",
|
||
"print(f\"Get the compressed sparse column (CSC) representation: {A.csc()}\")"
|
||
],
|
||
"metadata": {
|
||
"id": "OKbFiWKIzZVe"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**dtype and/or device conversion**"
|
||
],
|
||
"metadata": {
|
||
"id": "VzosM7i3yQPK"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"\n",
|
||
"B = A.to(device='cpu', dtype=torch.int32)\n",
|
||
"print(f\"Device sparse matrix is stored on: {B.device}\")\n",
|
||
"print(f\"Datatype of sparse matrix: {B.dtype}\")"
|
||
],
|
||
"metadata": {
|
||
"id": "y_RJihw-ypXp"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"Similar to pytorch, we also provide various fine-grained APIs ([Doc](https://docs.dgl.ai/en/latest/api/python/dgl.sparse_v0.html)) for dtype and/or device conversion."
|
||
],
|
||
"metadata": {
|
||
"id": "U26arLlJzfkN"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Diagonal Matrix\n",
|
||
"\n",
|
||
"Diagonal Matrix is a special type of Sparse Matrix, in which the entries outside the main diagonal are all zero.\n",
|
||
"\n",
|
||
"\n"
|
||
],
|
||
"metadata": {
|
||
"id": "EFe9ABRuWHqf"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### Initializing a DGL Diagonal Sparse Matrix\n",
|
||
"A DGL Diagonal Sparse Matrix can be initiate by `dglsp.diag()`.\n",
|
||
"\n",
|
||
"Identity Matrix is a special type of Diagonal Sparse Matrix, in which all the value on the diagonal are 1.0. Use `dglsp.identity()` to initiate a Diagonal Sparse Matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "1CeCoE2Fgl_x"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"D = dglsp.diag(val)\n",
|
||
"print(D)\n",
|
||
"\n",
|
||
"I = dglsp.identity(shape=(3, 3))\n",
|
||
"print(I)"
|
||
],
|
||
"metadata": {
|
||
"id": "9wzJNApahXAR"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Operations on Sparse Matrix\n",
|
||
"* Elementwise operations\n",
|
||
" * `A + B`\n",
|
||
" * `A - B`\n",
|
||
" * `A * B`\n",
|
||
" * `A / B`\n",
|
||
" * `A ** scalar`\n",
|
||
"* Broadcast operations\n",
|
||
" * `sp_<op>_v()`\n",
|
||
"* Reduce operations\n",
|
||
" * `reduce()`\n",
|
||
" * `sum()`\n",
|
||
" * `smax()`\n",
|
||
" * `smin()`\n",
|
||
" * `smean()`\n",
|
||
"* Matrix transformations\n",
|
||
" * `SparseMatrix.transpose()` or `SparseMatrix.T`\n",
|
||
" * `SparseMatrix.neg()`\n",
|
||
" * `SparseMatrix.inv()`\n",
|
||
"* Matrix multiplication\n",
|
||
" * `matmul()`\n",
|
||
" * `sddmm()`\n",
|
||
"\n",
|
||
"\n",
|
||
"*We are using dense format to print sparse matrix in this tutorial since it is more intuitive to read.*"
|
||
],
|
||
"metadata": {
|
||
"id": "Tjsapqp6zSFR"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### *Elementwise operations*"
|
||
],
|
||
"metadata": {
|
||
"id": "psvGwcIqYvC2"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**add(A, B), equivalent to A + B**\n",
|
||
"\n",
|
||
"Element-wise addition on two sparse matrices, returning a sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "39YJitpW-K9v"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A1:\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"\n",
|
||
"i = torch.tensor([[0, 1, 2],\n",
|
||
" [0, 2, 1]])\n",
|
||
"val = torch.tensor([4., 5., 6.])\n",
|
||
"A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A2:\")\n",
|
||
"print(A2.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D1 = dglsp.diag(val)\n",
|
||
"print(\"D1:\")\n",
|
||
"print(D1.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-4., -5., -6.])\n",
|
||
"D2 = dglsp.diag(val)\n",
|
||
"print(\"D2:\")\n",
|
||
"print(D2.to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 + A2:\")\n",
|
||
"print((A1 + A2).to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 + D1:\")\n",
|
||
"print((A1 + D1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 + D2:\")\n",
|
||
"print((D1 + D2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "pj3Ckx41-BSu"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**sub(A, B), equivalent to A - B**\n",
|
||
"\n",
|
||
"Element-wise substraction on two sparse matrices, returning a sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "i25N0JHUTUX9"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A1:\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"\n",
|
||
"i = torch.tensor([[0, 1, 2],\n",
|
||
" [0, 2, 1]])\n",
|
||
"val = torch.tensor([4., 5., 6.])\n",
|
||
"A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A2:\")\n",
|
||
"print(A2.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D1 = dglsp.diag(val)\n",
|
||
"print(\"D1:\")\n",
|
||
"print(D1.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-4., -5., -6.])\n",
|
||
"D2 = dglsp.diag(val)\n",
|
||
"print(\"D2:\")\n",
|
||
"print(D2.to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 - A2:\")\n",
|
||
"print((A1 - A2).to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 - D1:\")\n",
|
||
"print((A1 - D1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 - A1:\")\n",
|
||
"print((D1 - A1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 - D2:\")\n",
|
||
"print((D1 - D2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "GMxfz-cyT129"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**mul(A, B), equivalent to A * B**\n",
|
||
"\n",
|
||
"Element-wise multiplication on two sparse matrices or on a sparse matrix and a scalar, returning a sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "bg45jnq8T9EJ"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A1:\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"\n",
|
||
"i = torch.tensor([[0, 1, 2, 2],\n",
|
||
" [0, 2, 0, 1]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"\n",
|
||
"print(\"A2:\")\n",
|
||
"print(A2.to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 * 3:\")\n",
|
||
"print((A1 * 3).to_dense())\n",
|
||
"print(\"3 * A1:\")\n",
|
||
"print((3 * A1).to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 * A2\")\n",
|
||
"print((A1 * A2).to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D1 = dglsp.diag(val)\n",
|
||
"print(\"D1:\")\n",
|
||
"print(D1.to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 * A2\")\n",
|
||
"print((D1 * A2).to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-4., -5., -6.])\n",
|
||
"D2 = dglsp.diag(val)\n",
|
||
"print(\"D2:\")\n",
|
||
"print(D2.to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 * -2:\")\n",
|
||
"print((D1 * -2).to_dense())\n",
|
||
"print(\"-2 * D1:\")\n",
|
||
"print((-2 * D1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 * D2:\")\n",
|
||
"print((D1 * D2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "4PAITJqHUB8J"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**div(A, B), equivalent to A / B**\n",
|
||
"\n",
|
||
"Element-wise multiplication on two sparse matrices or on a sparse matrix and a scalar, returning a sparse matrix. If both `A` and `B` are sparse matrices, both of them must have the same sparsity. And the returned matrix has the same order of non-zero entries as `A`."
|
||
],
|
||
"metadata": {
|
||
"id": "Xb2RU6H4UBCs"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A1:\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"\n",
|
||
"i = torch.tensor([[1, 2, 1],\n",
|
||
" [0, 0, 2]])\n",
|
||
"val = torch.tensor([1., 3., 2.])\n",
|
||
"A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"\n",
|
||
"print(\"A1 / 2:\")\n",
|
||
"print((A1 / 2).to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 / A2\")\n",
|
||
"print((A1 / A2).to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D1 = dglsp.diag(val)\n",
|
||
"print(\"D1:\")\n",
|
||
"print(D1.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-4., -5., -6.])\n",
|
||
"D2 = dglsp.diag(val)\n",
|
||
"print(\"D2:\")\n",
|
||
"print(D2.to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 / D2:\")\n",
|
||
"print((D1 / D2).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 / 2:\")\n",
|
||
"print((D1 / 2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "TFB_UcmEUdr3"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**power(A, B), equivalent to A \\*\\* B**\n",
|
||
"\n",
|
||
"Element-wise power of a sparse matrix and a scalar, returning a sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "2lZbyTYUUgSi"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"print(\"A ** 3:\")\n",
|
||
"print((A ** 3).to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D = dglsp.diag(val)\n",
|
||
"print(\"D:\")\n",
|
||
"print(D.to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 ** 2:\")\n",
|
||
"print((D1 ** 2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "ox-XxCnuUqAy"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### *Broadcast operations*"
|
||
],
|
||
"metadata": {
|
||
"id": "VXBz4j5x_wQ4"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**sp_\\<op\\>_v(A, v)**\n",
|
||
"\n",
|
||
"Broadcast operations on a sparse matrix and a vector, returning a sparse matrix. `v` is broadcasted to the shape of `A` and then the operator is applied on the non-zero values of `A`. `<op>` can be add, sub, mul, and div. \n",
|
||
"\n",
|
||
"There are two cases regarding the shape of `v`:\n",
|
||
"\n",
|
||
"1. `v` is a vector of shape `(1, A.shape[1])` or `(A.shape[1])`. In this case, `v` is broadcasted on the row dimension of `A`.\n",
|
||
"\n",
|
||
"2. `v` is a vector of shape `(A.shape[0], 1)`. In this case, `v` is broadcasted on the column dimension of `A`."
|
||
],
|
||
"metadata": {
|
||
"id": "PtnyZdXHAZ6Z"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 0, 2], [0, 3, 2]])\n",
|
||
"val = torch.tensor([10, 20, 30])\n",
|
||
"A = dglsp.spmatrix(i, val, shape=(3, 4))\n",
|
||
"\n",
|
||
"v1 = torch.tensor([1, 2, 3, 4])\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"print(\"v1:\")\n",
|
||
"print(v1)\n",
|
||
"\n",
|
||
"print(\"sp_add_v(A, v1)\")\n",
|
||
"print(dglsp.sp_add_v(A, v1).to_dense())\n",
|
||
"\n",
|
||
"v2 = v1.reshape(1, -1)\n",
|
||
"print(\"v2:\")\n",
|
||
"print(v2)\n",
|
||
"\n",
|
||
"print(\"sp_add_v(A, v2)\")\n",
|
||
"print(dglsp.sp_add_v(A, v2).to_dense())\n",
|
||
"\n",
|
||
"v3 = torch.tensor([1, 2, 3]).reshape(-1, 1)\n",
|
||
"print(\"v3:\")\n",
|
||
"print(v3)\n",
|
||
"\n",
|
||
"print(\"sp_add_v(A, v3)\")\n",
|
||
"print(dglsp.sp_add_v(A, v3).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "xxf3s-uWBRR7"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### *Reduce operations*\n",
|
||
"\n",
|
||
"All DGL sparse reduce operations only consider non-zero elements. To distinguish them from dense PyTorch reduce operations that consider zero elements, we use name `smax`, `smin` and `smean` (`s` stands for sparse)."
|
||
],
|
||
"metadata": {
|
||
"id": "TQJJlctZjYPv"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"print(A.T.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"# O1, O2 will have the same value.\n",
|
||
"O1 = A.reduce(0, 'sum')\n",
|
||
"O2 = A.sum(0)\n",
|
||
"print(\"Reduce with reducer:sum along dim = 0:\")\n",
|
||
"print(O1)\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"# O3, O4 will have the same value.\n",
|
||
"O3 = A.reduce(0, 'smax')\n",
|
||
"O4 = A.smax(0)\n",
|
||
"print(\"Reduce with reducer:max along dim = 0:\")\n",
|
||
"print(O3)\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"# O5, O6 will have the same value.\n",
|
||
"O5 = A.reduce(0, 'smin')\n",
|
||
"O6 = A.smin(0)\n",
|
||
"print(\"Reduce with reducer:min along dim = 0:\")\n",
|
||
"print(O5)\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"# O7, O8 will have the same value.\n",
|
||
"O7 = A.reduce(0, 'smean')\n",
|
||
"O8 = A.smean(0)\n",
|
||
"print(\"Reduce with reducer:smean along dim = 0:\")\n",
|
||
"print(O7)\n",
|
||
"print(\"\")"
|
||
],
|
||
"metadata": {
|
||
"id": "GhS49Js1jW4b"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### *Matrix transformations*"
|
||
],
|
||
"metadata": {
|
||
"id": "kanwnB7LOQui"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"*Sparse Matrix*"
|
||
],
|
||
"metadata": {
|
||
"id": "NiiXso9elM2p"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"print(A.to_dense())\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"print(\"Get transpose of sparse matrix.\")\n",
|
||
"print(A.T.to_dense())\n",
|
||
"# Alias\n",
|
||
"# A.transpose()\n",
|
||
"# A.t()\n",
|
||
"print(\"\")\n",
|
||
"\n",
|
||
"print(\"Get a sparse matrix with the negation of the original nonzero values.\")\n",
|
||
"print(A.neg().to_dense())\n",
|
||
"print(\"\")"
|
||
],
|
||
"metadata": {
|
||
"id": "qJcmZHmf-oTY"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### *Matrix multiplication*"
|
||
],
|
||
"metadata": {
|
||
"id": "4uQlDFb0Uzto"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**matmul(A, B), equivalent to A @ B**\n",
|
||
"\n",
|
||
"Matrix multiplication on sparse matrices and/or dense matrix. There are two cases as follows."
|
||
],
|
||
"metadata": {
|
||
"id": "THWE30v6WpAk"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**SparseMatrix @ SparseMatrix -> SparseMatrix:**\n",
|
||
"\n",
|
||
"For a $L \\times M$ sparse matrix A and a $M \\times N$ sparse matrix B, the shape of `A @ B` will be $L \\times N$ sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "VxyykR-vX7lF"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A1 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A1:\")\n",
|
||
"print(A1.to_dense())\n",
|
||
"\n",
|
||
"i = torch.tensor([[0, 1, 2],\n",
|
||
" [0, 2, 1]])\n",
|
||
"val = torch.tensor([4., 5., 6.])\n",
|
||
"A2 = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A2:\")\n",
|
||
"print(A2.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D1 = dglsp.diag(val)\n",
|
||
"print(\"D1:\")\n",
|
||
"print(D1.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-4., -5., -6.])\n",
|
||
"D2 = dglsp.diag(val)\n",
|
||
"print(\"D2:\")\n",
|
||
"print(D2.to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 @ A2:\")\n",
|
||
"print((A1 @ A2).to_dense())\n",
|
||
"\n",
|
||
"print(\"A1 @ D1:\")\n",
|
||
"print((A1 @ D1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 @ A1:\")\n",
|
||
"print((D1 @ A1).to_dense())\n",
|
||
"\n",
|
||
"print(\"D1 @ D2:\")\n",
|
||
"print((D1 @ D2).to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "XRDFC2rOYQM4"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**SparseMatrix @ Tensor -> Tensor:**\n",
|
||
"\n",
|
||
"For a $L \\times M$ sparse matrix A and a $M \\times N$ dense matrix B, the shape of `A @ B` will be $L \\times N$ dense matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "g13fG8nvaVOt"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"val = torch.tensor([-1., -2., -3.])\n",
|
||
"D = dglsp.diag(val)\n",
|
||
"print(\"D:\")\n",
|
||
"print(D.to_dense())\n",
|
||
"\n",
|
||
"X = torch.tensor([[11., 22.], [33., 44.], [55., 66.]])\n",
|
||
"print(\"X:\")\n",
|
||
"print(X)\n",
|
||
"\n",
|
||
"print(\"A @ X:\")\n",
|
||
"print(A @ X)\n",
|
||
"\n",
|
||
"print(\"D @ X:\")\n",
|
||
"print(D @ X)"
|
||
],
|
||
"metadata": {
|
||
"id": "FcQ-CnqdlgWF"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"This operator also supports batched sparse-dense matrix multiplication. The sparse matrix A should have shape $L \\times M$, where the non-zero values are vectors of length $K$. The dense matrix B should have shape $M \\times N \\times K$. The output is a dense matrix of shape $L \\times N \\times K$."
|
||
],
|
||
"metadata": {
|
||
"id": "_KZiULLbmEZE"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [0, 2, 0]])\n",
|
||
"val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
|
||
"A = dglsp.spmatrix(i, val, shape=(3, 3))\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"X = torch.tensor([[[1., 1.], [1., 2.]],\n",
|
||
" [[1., 3.], [1., 4.]],\n",
|
||
" [[1., 5.], [1., 6.]]])\n",
|
||
"print(\"X:\")\n",
|
||
"print(X)\n",
|
||
"\n",
|
||
"print(\"A @ X:\")\n",
|
||
"print(A @ X)"
|
||
],
|
||
"metadata": {
|
||
"id": "ZUzXQk7Ab2wG"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**Sampled-Dense-Dense Matrix Multiplication (SDDMM)**\n",
|
||
"\n",
|
||
"``sddmm`` matrix-multiplies two dense matrices X1 and X2, then elementwise-multiplies the result with sparse matrix A at the nonzero locations. This is designed for sparse matrix with scalar values.\n",
|
||
"\n",
|
||
"$$out = (X_1 @ X_2) * A$$\n",
|
||
"\n",
|
||
"For a $L \\times N$ sparse matrix A, a $L \\times M$ dense matrix X1 and a $M \\times N$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N$ sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "qO_8f_vhPKtf"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [2, 3, 3]])\n",
|
||
"val = torch.tensor([1., 2., 3.])\n",
|
||
"A = dglsp.spmatrix(i, val, (3, 4))\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"X1 = torch.randn(3, 5)\n",
|
||
"X2 = torch.randn(5, 4)\n",
|
||
"print(\"X1:\")\n",
|
||
"print(X1)\n",
|
||
"print(\"X2:\")\n",
|
||
"print(X2)\n",
|
||
"\n",
|
||
"O = dglsp.sddmm(A, X1, X2)\n",
|
||
"print(\"dglsp.sddmm(A, X1, X2):\")\n",
|
||
"print(O.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "3ZIFV0TgPhwH"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"This operator also supports batched sampled-dense-dense matrix multiplication. For a $L \\times N$ sparse matrix A with non-zero vector values of length $𝐾$, a $L \\times M \\times K$ dense matrix X1 and a $M \\times N \\times K$ dense matrix X2, `sddmm(A, X1, X2)` will be a $L \\times N \\times K$ sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "RmNmXU_ZqyF7"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[1, 1, 2],\n",
|
||
" [2, 3, 3]])\n",
|
||
"val = torch.tensor([[1., 1.], [2., 2.], [3., 3.]])\n",
|
||
"A = dglsp.spmatrix(i, val, (3, 4))\n",
|
||
"print(\"A:\")\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"X1 = torch.randn(3, 5, 2)\n",
|
||
"X2 = torch.randn(5, 4, 2)\n",
|
||
"print(\"X1:\")\n",
|
||
"print(X1)\n",
|
||
"print(\"X2:\")\n",
|
||
"print(X2)\n",
|
||
"\n",
|
||
"O = dglsp.sddmm(A, X1, X2)\n",
|
||
"print(\"dglsp.sddmm(A, X1, X2):\")\n",
|
||
"print(O.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "DuSAjamyrIO_"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Non-linear activation functions"
|
||
],
|
||
"metadata": {
|
||
"id": "fVkbTT28ZzPr"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### Element-wise functions\n",
|
||
"\n",
|
||
"Most activation functions are element-wise and can be further grouped into two categories:\n",
|
||
"\n",
|
||
"**Sparse-preserving functions** such as `sin()`, `tanh()`, `sigmoid()`, `relu()`, etc. You can directly apply them on the `val` tensor of the sparse matrix and then recreate a new matrix of the same sparsity using `val_like`."
|
||
],
|
||
"metadata": {
|
||
"id": "XuaNdFO7XG2r"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.randn(4)\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"print(\"Apply tanh.\")\n",
|
||
"A_new = dglsp.val_like(A, torch.tanh(A.val))\n",
|
||
"print(A_new.to_dense())"
|
||
],
|
||
"metadata": {
|
||
"id": "GZkCJJ0TX0cI"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"**Non-sparse-preserving functions** such as `exp()`, `cos()`, etc. You can first convert the sparse matrix to dense before applying the functions."
|
||
],
|
||
"metadata": {
|
||
"id": "i92lhMEnYas3"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.randn(4)\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"print(A.to_dense())\n",
|
||
"\n",
|
||
"print(\"Apply exp.\")\n",
|
||
"A_new = A.to_dense().exp()\n",
|
||
"print(A_new)"
|
||
],
|
||
"metadata": {
|
||
"id": "sroJpzRNYZq5"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"### Softmax\n",
|
||
"\n",
|
||
"Apply row-wise softmax to the nonzero entries of the sparse matrix."
|
||
],
|
||
"metadata": {
|
||
"id": "y8OQZReVXpo3"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 1, 1, 2],\n",
|
||
" [1, 0, 2, 0]])\n",
|
||
"val = torch.tensor([1., 2., 3., 4.])\n",
|
||
"A = dglsp.spmatrix(i, val)\n",
|
||
"\n",
|
||
"print(A.softmax())\n",
|
||
"print(\"In dense format:\")\n",
|
||
"print(A.softmax().to_dense())\n",
|
||
"print(\"\\n\")"
|
||
],
|
||
"metadata": {
|
||
"id": "CQaKgzCJULjt"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Exercise \\#1\n",
|
||
"\n",
|
||
"*Let's test what you've learned. Feel free to [](https://colab.research.google.com/github/dmlc/dgl/blob/master/notebooks/sparse/quickstart.ipynb).*\n",
|
||
"\n",
|
||
"Given a sparse symmetrical adjacency matrix $A$, calculate its symmetrically normalized adjacency matrix: $$norm = \\bar{D}^{-\\frac{1}{2}}\\bar{A}\\bar{D}^{-\\frac{1}{2}}$$\n",
|
||
"\n",
|
||
"Where $\\bar{A} = A + I$, $I$ is the identity matrix, and $\\bar{D}$ is the diagonal node degree matrix of $\\bar{A}$."
|
||
],
|
||
"metadata": {
|
||
"id": "1iBNlJVYz3zi"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"i = torch.tensor([[0, 0, 1, 1, 2, 2, 3],\n",
|
||
" [1, 3, 2, 5, 3, 5, 4]])\n",
|
||
"asym_A = dglsp.spmatrix(i, shape=(6, 6))\n",
|
||
"# Step 1: create symmetrical adjacency matrix A from asym_A.\n",
|
||
"# A =\n",
|
||
"\n",
|
||
"# Step 2: calculate A_hat from A.\n",
|
||
"# A_hat =\n",
|
||
"\n",
|
||
"# Step 3: diagonal node degree matrix of A_hat\n",
|
||
"# D_hat =\n",
|
||
"\n",
|
||
"# Step 4: calculate the norm from D_hat and A_hat.\n",
|
||
"# norm = "
|
||
],
|
||
"metadata": {
|
||
"id": "0dDhfbJo0ByV"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"source": [
|
||
"## Exercise \\#2\n",
|
||
"\n",
|
||
"Let's implement a simplified version of the Graph Attention Network (GAT) layer.\n",
|
||
"\n",
|
||
"A GAT layer has two inputs: the adjacency matrix $A$ and the node input features $X$. The idea of GAT layer is to update each node's representation with a weighted average of the node's own representation and its neighbors' representations. In particular, when computing the output for node $i$, the GAT layer does the following:\n",
|
||
"1. Compute the scores $S_{ij}$ representing the attention logit from neighbor $j$ to node $i$. $S_{ij}$ is a function of $i$ and $j$'s input features $X_i$ and $X_j$: $$S_{ij} = LeakyReLU(X_i^\\top v_1 + X_j^\\top v_2)$$, where $v_1$ and $v_2$ are trainable vectors.\n",
|
||
"2. Compute a softmax attention $R_{ij} = \\exp S_{ij} / \\left( \\sum_{j' \\in \\mathcal{N}_i} s_{ij'} \\right)$, where $\\mathcal{N}_j$ means the neighbors of $j$. This means that $R$ is a row-wise softmax attention of $S$.\n",
|
||
"3. Compute the weighted average $H_i = \\sum_{j' : j' \\in \\mathcal{N}_i} R_{j'} X_{j'} W$, where $W$ is a trainable matrix.\n",
|
||
"\n",
|
||
"The following code defined all the parameters you need but only completes step 1. Could you implement step 2 and step 3?"
|
||
],
|
||
"metadata": {
|
||
"id": "yfEVQBUuI-cE"
|
||
}
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"import torch.nn as nn\n",
|
||
"import torch.nn.functional as F\n",
|
||
"\n",
|
||
"class SimplifiedGAT(nn.Module):\n",
|
||
" def __init__(self, in_size, out_size):\n",
|
||
" super().__init__()\n",
|
||
"\n",
|
||
" self.W = nn.Parameter(torch.randn(in_size, out_size))\n",
|
||
" self.v1 = nn.Parameter(torch.randn(in_size))\n",
|
||
" self.v2 = nn.Parameter(torch.randn(in_size))\n",
|
||
"\n",
|
||
" def forward(self, A, X):\n",
|
||
" # A: A sparse matrix with size (N, N). A[i, j] represent the edge from j to i.\n",
|
||
" # X: A dense matrix with size (N, D)\n",
|
||
" # Step 1: compute S[i, j]\n",
|
||
" Xv1 = X @ self.v1\n",
|
||
" Xv2 = X @ self.v2\n",
|
||
" s = F.leaky_relu(Xv1[A.col] + Xv2[A.row])\n",
|
||
" S = dglsp.val_like(A, s)\n",
|
||
"\n",
|
||
" # Step 2: compute R[i, j] which is the row-wise attention of $S$.\n",
|
||
" # EXERCISE: replace the statement below.\n",
|
||
" R = S\n",
|
||
"\n",
|
||
" # Step 3: compute H.\n",
|
||
" # EXERCISE: replace the statement below.\n",
|
||
" H = X\n",
|
||
"\n",
|
||
" return H"
|
||
],
|
||
"metadata": {
|
||
"id": "pYrgSxq6La5c"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"source": [
|
||
"# Test:\n",
|
||
"# Let's use the symmetric A created above.\n",
|
||
"X = torch.randn(6, 20)\n",
|
||
"module = SimplifiedGAT(20, 10)\n",
|
||
"Y = module(A, X)"
|
||
],
|
||
"metadata": {
|
||
"id": "qjcXiidYCqGK"
|
||
},
|
||
"execution_count": null,
|
||
"outputs": []
|
||
}
|
||
]
|
||
}
|