wehub-resource-sync
1147 lines
39 KiB
Plaintext
1147 lines
39 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bee64cb2-8555-4239-babc-8f0d249bce19",
|
||
"metadata": {},
|
||
"source": [
|
||
"<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/extras/pytorch_cheatsheet.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a> "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b721b6b3-bf94-4f8f-9aae-9fa119962b1d",
|
||
"metadata": {},
|
||
"source": [
|
||
"# PyTorch Cheatsheet\n",
|
||
"\n",
|
||
"Some of the most commonly used commands/setups in PyTorch.\n",
|
||
"\n",
|
||
"> **Note:** One of the best ways to get help for PyTorch specific functions and use cases is to search \"pytorch how to make a convolutional neural network\" or \"pytorch transformer layers\" or \"pytorch loss functions\". I do this regularly."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0ee9c36f-3fd9-42b8-a5b6-a5a4de1e5cdd",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Imports\n",
|
||
"\n",
|
||
"You can install PyTorch on various platforms via the [PyTorch installation page](https://pytorch.org/get-started/locally/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "98cf25ad-8e15-4692-a496-dc69b83a2f00",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"PyTorch version: 1.13.1\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"import torch\n",
|
||
"\n",
|
||
"# Check the version\n",
|
||
"print(f\"PyTorch version: {torch.__version__}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "9e369e44-8cea-4171-87f4-93bf421155c1",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"<class 'torch.nn.modules.linear.Linear'>\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Can also import the common abbreviation \"nn\" for \"Neural Networks\"\n",
|
||
"from torch import nn\n",
|
||
"\n",
|
||
"# Almost everything in PyTorch is called a \"Module\" (you build neural networks by stacking together Modules)\n",
|
||
"this_is_a_module = nn.Linear(in_features=1,\n",
|
||
" out_features=1)\n",
|
||
"print(type(this_is_a_module))"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d89b206a-6869-4d33-b313-e952aef82adf",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Data imports\n",
|
||
"\n",
|
||
"Since most of machine learning is finding patterns in data, it's good to know how to work with datasets in PyTorch."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "5a6cf16f-f713-4a5a-8666-a622695eb612",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Import PyTorch Dataset (you can store your data here) and DataLoader (you can load your data here)\n",
|
||
"from torch.utils.data import Dataset, DataLoader"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "58418977-bdb1-4dfb-858c-3d99e5d0ef77",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Creating Tensors\n",
|
||
"\n",
|
||
"One of the main use cases of PyTorch is for accelerated deep learning computing.\n",
|
||
"\n",
|
||
"And deep learning usually involves the manipulation of large tensors (big, multi-dimensional collections of numbers).\n",
|
||
"\n",
|
||
"PyTorch has a number of methods to create tensors.\n",
|
||
"\n",
|
||
"> **Note:** For a more extensive overview of creating tensors with PyTorch, see [00. PyTorch Fundamentals](https://www.learnpytorch.io/00_pytorch_fundamentals/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "ce6c9952-5783-4ade-8c92-06b4393139ef",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a single number tensor (scalar)\n",
|
||
"scalar = torch.tensor(7)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "f3f4f286-a8e6-4db0-8d5e-0c7aaf3b7c2b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a random tensor\n",
|
||
"random_tensor = torch.rand(size=(3, 4)) # this will create a tensor of size 3x4 but you can manipulate the shape however you want"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "bf17bde1-c03d-4f7c-ab2a-d24f59161cf6",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Multiply two random tensors\n",
|
||
"random_tensor_1 = torch.rand(size=(3, 4))\n",
|
||
"random_tensor_2 = torch.rand(size=(3, 4))\n",
|
||
"random_tensor_3 = random_tensor_1 * random_tensor_2 # PyTorch has support for most math operators in Python (+, *, -, /)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3f51efe1-609a-40c0-94a2-1f6c816599ef",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Domain Libraries\n",
|
||
"\n",
|
||
"Depending on the specific problem you're working on, PyTorch has several domain libraries.\n",
|
||
"\n",
|
||
"- **[TorchVision](https://pytorch.org/vision/stable/index.html)** — PyTorch’s resident computer vision library. \n",
|
||
"- **[TorchText](https://pytorch.org/text/stable/index.html)** — PyTorch’s in-built domain library for text.\n",
|
||
"- [**TorchAudio**](https://pytorch.org/audio/stable/index.html) — PyTorch’s domain library for everything audio. \n",
|
||
"- **[TorchRec](https://pytorch.org/torchrec/)** — PyTorch’s newest in-built domain library for powering recommendation engines with deep learning."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "966a5d29-d2f4-4089-b3a7-6b400f695d20",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Computer Vision \n",
|
||
"\n",
|
||
"> **Note:** For an in-depth overview of computer vision in PyTorch, see [03. PyTorch Computer Vision](https://www.learnpytorch.io/03_pytorch_computer_vision/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "5353bfa2-d950-4f23-98c3-d29ff7c85e85",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Base computer vision library\n",
|
||
"import torchvision\n",
|
||
"\n",
|
||
"# Other components of TorchVision (premade datasets, pretrained models and image transforms)\n",
|
||
"from torchvision import datasets, models, transforms "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d044f35e-068f-4664-a16c-b3150a842ce2",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Text and Natural Language Processing (NLP)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "328b5026-fb0e-42f6-a50f-e4171a379c86",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Base text and natural language processing library\n",
|
||
"import torchtext\n",
|
||
"\n",
|
||
"# Other components of TorchText (premade datasets, pretrained models and text transforms)\n",
|
||
"from torchtext import datasets, models, transforms"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "2409d640-a768-4423-a37c-3f4f2472b319",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Audio and Speech"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "956edd5a-291b-41d5-b7fa-507a171171dc",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Base audio and speech processing library\n",
|
||
"import torchaudio\n",
|
||
"\n",
|
||
"# Other components of TorchAudio (premade datasets, pretrained models and text transforms)\n",
|
||
"from torchaudio import datasets, models, transforms"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0cf19cae-f18b-4bb9-af99-56725a739df7",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Recommendation systems\n",
|
||
"\n",
|
||
"> **Note:** This library is currently in beta release, see the [GitHub page for installation](https://github.com/pytorch/torchrec#installation). "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "ddb24f63-13c9-406c-9608-f66042a99f87",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# # Base recommendation system library \n",
|
||
"# import torchrec\n",
|
||
"\n",
|
||
"# # Other components of TorchRec\n",
|
||
"# from torchrec import datasets, models"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f0dc7202-271a-4473-94fc-4945319bda37",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Device-agnostic code (using PyTorch on CPU, GPU or MPS)\n",
|
||
"\n",
|
||
"Much of deep learning involves computing on tensors. \n",
|
||
"\n",
|
||
"Computing on tensors generally happens much faster on GPUs (graphics processing units, typically from NVIDIA) than CPUs (computer processing units).\n",
|
||
"\n",
|
||
"MPS stands for \"Metal Performance Shader\" which is Apple's GPU (M1, M1 Pro, M2 etc).\n",
|
||
"\n",
|
||
"It is advised to perform training on the fastest piece of hardware you have available, which will generally be: NVIDIA GPU (`\"cuda\"`) > MPS device (`\"mps\"`) > CPU (`\"cpu\"`).\n",
|
||
"\n",
|
||
"* For more on seeing how to get [PyTorch to run on NVIDIA GPU (with CUDA)](https://pytorch.org/docs/stable/cuda.html), see [00. PyTorch Fundamentals section 2: getting PyTorch to run on the GPU](https://www.learnpytorch.io/00_pytorch_fundamentals/#2-getting-pytorch-to-run-on-the-gpu).\n",
|
||
"* For more on running PyTorch using an MPS backend (running PyTorch on Mac GPUs) [see the PyTorch documentation](https://pytorch.org/docs/stable/notes/mps.html). \n",
|
||
"\n",
|
||
"> **Note:** It is advised to setup device-agnostic code at the start of your workflow."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "081f561d-63f7-4609-86eb-868750eb8b8a",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Using device: mps\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Setup device-agnostic code \n",
|
||
"if torch.cuda.is_available():\n",
|
||
" device = \"cuda\" # NVIDIA GPU\n",
|
||
"elif torch.backends.mps.is_available():\n",
|
||
" device = \"mps\" # Apple GPU\n",
|
||
"else:\n",
|
||
" device = \"cpu\" # Defaults to CPU if NVIDIA GPU/Apple GPU aren't available\n",
|
||
"\n",
|
||
"print(f\"Using device: {device}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "208f1f37-470f-486a-9a71-b7128ae295a9",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Sending a tensor to target device\n",
|
||
"\n",
|
||
"You can move objects (models and tensors) in PyTorch to different devices via the `.to(\"device_name\")` method."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "7dc6c29a-5d0a-45e1-9dde-547356e1366e",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"cpu\n",
|
||
"mps:0\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create a tensor \n",
|
||
"x = torch.tensor([1, 2, 3]) \n",
|
||
"print(x.device) # defaults to CPU \n",
|
||
"\n",
|
||
"# Send tensor to target device\n",
|
||
"x = x.to(device)\n",
|
||
"print(x.device) "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9d27d062-281b-4628-9374-c276640aceb8",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Setting random seeds\n",
|
||
"\n",
|
||
"A lot of machine learning and deep learning involves taking random numbers in tensors and then shaping those random numbers to find/represent patterns in real data. \n",
|
||
"\n",
|
||
"However, sometimes you'll want \"reproducible\" randomness.\n",
|
||
"\n",
|
||
"To do so, you can set the random seeds, see [Reproducibility (trying to take the random out of random)](https://www.learnpytorch.io/00_pytorch_fundamentals/#reproducibility-trying-to-take-the-random-out-of-random) for more."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "de557ce2-60df-4945-a57a-a8dc5e7de756",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Tensor A:\n",
|
||
"tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
|
||
" [0.3904, 0.6009, 0.2566, 0.7936],\n",
|
||
" [0.9408, 0.1332, 0.9346, 0.5936]])\n",
|
||
"\n",
|
||
"Tensor B:\n",
|
||
"tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
|
||
" [0.3904, 0.6009, 0.2566, 0.7936],\n",
|
||
" [0.9408, 0.1332, 0.9346, 0.5936]])\n",
|
||
"\n",
|
||
"Does Tensor A equal Tensor B? (anywhere)\n"
|
||
]
|
||
},
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"tensor([[True, True, True, True],\n",
|
||
" [True, True, True, True],\n",
|
||
" [True, True, True, True]])"
|
||
]
|
||
},
|
||
"execution_count": 13,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"import torch\n",
|
||
"\n",
|
||
"# Set the random seed (you can set this to any number you like, it will \"flavour\"\n",
|
||
"# the randomness with that number.\n",
|
||
"torch.manual_seed(42)\n",
|
||
"\n",
|
||
"# Create two random tensors\n",
|
||
"random_tensor_A = torch.rand(3, 4)\n",
|
||
"\n",
|
||
"torch.manual_seed(42) # set the seed again (try commenting this out and see what happens)\n",
|
||
"random_tensor_B = torch.rand(3, 4)\n",
|
||
"\n",
|
||
"print(f\"Tensor A:\\n{random_tensor_A}\\n\")\n",
|
||
"print(f\"Tensor B:\\n{random_tensor_B}\\n\")\n",
|
||
"print(f\"Does Tensor A equal Tensor B? (anywhere)\")\n",
|
||
"random_tensor_A == random_tensor_B"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e3ea594e-6ae6-4637-97bb-e52cd537df64",
|
||
"metadata": {},
|
||
"source": [
|
||
"You can also set the random seed on the GPU (CUDA devices)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "b2499098-9311-40b2-8ff3-63d433064aaa",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Set random seed on GPU\n",
|
||
"torch.cuda.manual_seed(42)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3efd5fba-9578-481b-850e-a968ade7700a",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Neural Networks\n",
|
||
"\n",
|
||
"PyTorch has a very comprehensive library of pre-built neural network components (many of these are referred to as \"modules\" in the PyTorch ecosystem).\n",
|
||
"\n",
|
||
"At a fundamental level neural networks are stacks of *layers*. Each of these layers performs some kind of operation on an input and produces an output.\n",
|
||
"\n",
|
||
"How these layers stack together will depend on the problem you're working on.\n",
|
||
"\n",
|
||
"One of the most active areas of research in machine learning is how to stack neural network layers together (and the best answer to this is constantly changing). \n",
|
||
"\n",
|
||
"The vast majority of neural network components in PyTorch are contained within the [`torch.nn` package](https://pytorch.org/docs/stable/nn.html) (`nn` is short for neural networks)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "888b039b-c4ad-4f47-bad7-75ac8ca7db12",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from torch import nn"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "497528d9-9fef-4323-8438-b7a298ed7ea3",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Linear layers\n",
|
||
"\n",
|
||
"PyTorch has several in-built [linear layers](https://pytorch.org/docs/stable/nn.html#linear-layers)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "79047b27-3752-4e36-b7a9-a8e95bac20c4",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a linear layer with 10 in features and out features\n",
|
||
"linear_layer = nn.Linear(in_features=10,\n",
|
||
" out_features=10)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "4972c605-39b4-43e4-aba4-02d9527cbbde",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create an Identity layer\n",
|
||
"identity_layer = nn.Identity()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9115ee4f-67a1-4adb-92e1-7604ccbb4b62",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Convolutional Layers (for making Convolutional Neural Networks or CNN's)\n",
|
||
"\n",
|
||
"PyTorch has [several in-built convolutional layers](https://pytorch.org/docs/stable/nn.html#convolution-layers).\n",
|
||
"\n",
|
||
"Naming of convolutional layers usually follows `torch.nn.ConvXd` where `X` can be a value of `1`, `2` or `3`.\n",
|
||
"\n",
|
||
"The `X` value represents the number of dimensions the convolution will operate over, for example, `1` for singular dimension text, `2` for two dimension images (height x width) and `3` for 3D objects such as video (video is considered a series of images with a time dimension, height x width x time).\n",
|
||
"\n",
|
||
"> **Note:** You can see more on building convolutional neural networks for computer vision with PyTorch in [03. PyTorch Computer Vision section 7.2: building a convolutional neural network (CNN)](https://www.learnpytorch.io/03_pytorch_computer_vision/#7-model-2-building-a-convolutional-neural-network-cnn)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "882a8c21-cc87-4ebe-9a03-68f9765647c2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a Conv1d layer (often used for text with a singular dimension)\n",
|
||
"conv1d = nn.Conv1d(in_channels=1,\n",
|
||
" out_channels=10,\n",
|
||
" kernel_size=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "93d96033-75e4-4695-ad39-c488bb0a5038",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a Conv2d layer (often used for images with Height x Width dimensions)\n",
|
||
"conv2d = nn.Conv2d(in_channels=3, # 3 channels for color images (red, green, blue)\n",
|
||
" out_channels=10,\n",
|
||
" kernel_size=3) "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "b0b697f0-b80b-4dc7-a7d9-b771dad6c171",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a Conv3d layer (often used for video with Height x Width x Time dimensions)\n",
|
||
"conv3d = nn.Conv3d(in_channels=3,\n",
|
||
" out_channels=10,\n",
|
||
" kernel_size=3)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6042dabb-9f18-4cd1-916b-b0d864048b7e",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Transformer Layers (for making Transformer models)\n",
|
||
"\n",
|
||
"PyTorch has in-built Transformer layers as described in the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n",
|
||
"\n",
|
||
"Using in-built PyTorch Transformer layers has the benefit of potential speedups thanks to [PyTorch's BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/).\n",
|
||
"\n",
|
||
"> **Note:** You can see the use of PyTorch's in-built Transformer layers to build a Vision Transformer in [08. PyTorch Paper Replicating](https://www.learnpytorch.io/08_pytorch_paper_replicating/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"id": "0bcab8b8-defc-4d8c-8d75-c7826f200dd1",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a Transformer model (model based on the paper \"Attention Is All You Need\" - https://arxiv.org/abs/1706.03762)\n",
|
||
"transformer_model = nn.Transformer()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"id": "ba384ef7-d6f4-4969-80da-f477634ea96f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a single Transformer encoder cell\n",
|
||
"transformer_encoder = nn.TransformerEncoderLayer(d_model=768, # embedding dimension\n",
|
||
" nhead=12) # number of attention heads"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 23,
|
||
"id": "6e761141-0967-4804-a71b-03994132f865",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Stack together Transformer encoder cells\n",
|
||
"transformer_encoder_stack = nn.TransformerEncoder(encoder_layer=transformer_encoder, # from above\n",
|
||
" num_layers=6) # 6 Transformer encoders stacked on top of each other"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 24,
|
||
"id": "afdc3a0f-b4c4-47a2-ab29-999e66b5ed08",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a single Transformer decoder cell\n",
|
||
"transformer_decoder = nn.TransformerDecoderLayer(d_model=768,\n",
|
||
" nhead=12)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 25,
|
||
"id": "0bf68ba3-2f4b-454f-a5fe-3707bf52fa9d",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Stack together Transformer decoder cells\n",
|
||
"transformer_decoder_stack = nn.TransformerDecoder(decoder_layer=transformer_decoder, # from above\n",
|
||
" num_layers=6) # 6 Transformer decoders stacked on top of each other"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a6397f85-2263-47b1-b720-c3d46bf0eaef",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Recurrent Layers (for making Recurrent Neural Networks or RNN's)\n",
|
||
"\n",
|
||
"PyTorch has in-built support for [Recurrent Neural Network layers](https://pytorch.org/docs/stable/nn.html#recurrent-layers) such as [long short-term memory (LSTM)](https://en.wikipedia.org/wiki/Long_short-term_memory) and [gated recurrent unit (GRU)](https://en.wikipedia.org/wiki/Gated_recurrent_unit).\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 26,
|
||
"id": "aabbfdc0-3673-42c8-b075-62664cc4be85",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a single LSTM cell\n",
|
||
"lstm_cell = nn.LSTMCell(input_size=10, # can adjust as necessary\n",
|
||
" hidden_size=10) # can adjust as necessary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 27,
|
||
"id": "df67845c-55b5-4e67-9058-699075644b7c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Stack together LSTM cells\n",
|
||
"lstm_stack = nn.LSTM(input_size=10,\n",
|
||
" hidden_size=10,\n",
|
||
" num_layers=3) # 3 single LSTM cells stacked on top of each other"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 28,
|
||
"id": "e421579e-ab99-4b5f-be73-2a7f11f70d36",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a single GRU cell\n",
|
||
"gru_cell = nn.GRUCell(input_size=10, # can adjust as necessary\n",
|
||
" hidden_size=10) # can adjust as necessary"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 29,
|
||
"id": "083284c1-501a-466a-b4e9-901a079f958c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Stack together GRU cells\n",
|
||
"gru_stack = nn.GRU(input_size=10, \n",
|
||
" hidden_size=10,\n",
|
||
" num_layers=3) # 3 single GRU cells stacked on top of each other "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5087611c-aae7-41fe-8397-03e01c885ade",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Activation Functions \n",
|
||
"\n",
|
||
"[Activation functions](https://en.wikipedia.org/wiki/Activation_function) often go between layers in a neural network to add non-linear (non-straight) capabilities to linear (straight) functions.\n",
|
||
"\n",
|
||
"In essence, a neural network is often comprised of a large amount of linear and non-linear functions.\n",
|
||
"\n",
|
||
"PyTorch has [several non-linear activation functions](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity) built into `torch.nn`.\n",
|
||
"\n",
|
||
"Some of the most common are:\n",
|
||
"* `nn.ReLU` - also known as [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)).\n",
|
||
"* `nn.Sigmoid` - also known as the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function).\n",
|
||
"* `nn.Softmax` - also known as the [softmax function](https://en.wikipedia.org/wiki/Softmax_function).\n",
|
||
"\n",
|
||
"> **Note:** See [02. PyTorch Neural Network Classification section 6: non-linearity, the missing piece](https://www.learnpytorch.io/02_pytorch_classification/#6-the-missing-piece-non-linearity) for more."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 30,
|
||
"id": "38581ec6-ed57-4c2f-89bb-f99193939cac",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# ReLU\n",
|
||
"relu = nn.ReLU()\n",
|
||
"\n",
|
||
"# Sigmoid\n",
|
||
"sigmoid = nn.Sigmoid()\n",
|
||
"\n",
|
||
"# Softmax\n",
|
||
"softmax = nn.Softmax()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bf38e1c3-1159-439d-ad3b-0db1b7fa3d12",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Loss Functions\n",
|
||
"\n",
|
||
"A loss function measures how *wrong* your model is. As in, how far are its predictions off where they should be.\n",
|
||
"\n",
|
||
"Ideally, with training, data and an optimization function, this loss value goes as low as possible.\n",
|
||
"\n",
|
||
"Loss functions in PyTorch (and deep learning in general) are also often referred to as: criterion, cost function.\n",
|
||
"\n",
|
||
"PyTorch has [several loss functions](https://pytorch.org/docs/stable/nn.html#loss-functions) built into `torch.nn`.\n",
|
||
"\n",
|
||
"And some of the most common are:\n",
|
||
"* [`nn.L1Loss`](https://pytorch.org/docs/stable/generated/torch.nn.L1Loss.html#torch.nn.L1Loss) - also referred to as MAE or [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (this loss is often used for regression problems or predicting a number such as the price of houses).\n",
|
||
"* [`nn.MSELoss`](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) - also referred to as L2Loss or [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error) (this loss is often used for regression problems or predicting a number such as the price of houses).\n",
|
||
"* [`nn.BCEWithLogitsLoss`](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss) - also known as [binary cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) this loss function is often used for binary classification problems (classifying something as one thing or another).\n",
|
||
"* [`nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss) - this loss function is often used for multi-class classification problems (classifying something as one thing or another)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 31,
|
||
"id": "7ffe3146-759e-4f2f-aeea-ec1ecb506ad2",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# L1Loss\n",
|
||
"loss_fn = nn.L1Loss() # also known as MAE or mean absolute error\n",
|
||
"\n",
|
||
"# MSELoss\n",
|
||
"loss_fn = nn.MSELoss() # also known as MSE or mean squared error\n",
|
||
"\n",
|
||
"# Binary cross entropy (for binary classification problems)\n",
|
||
"loss_fn = nn.BCEWithLogitsLoss()\n",
|
||
"\n",
|
||
"# Cross entropy (for multi-class classification problems)\n",
|
||
"loss_fn = nn.CrossEntropyLoss()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e5746cf2-2ff9-4621-a681-df36cac3beb1",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Optimizers \n",
|
||
"\n",
|
||
"An optimizer's job is to change the neural network weights in such a way that it reduces the loss function value.\n",
|
||
"\n",
|
||
"PyTorch has [several optimization functions](https://pytorch.org/docs/stable/optim.html) built into the `torch.optim` module. \n",
|
||
"\n",
|
||
"Two of the main optimizer functions include:\n",
|
||
"* [`torch.optim.SGD(lr=0.1, params=model.parameters())`](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD) - SGD also known as [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (`lr` stands for \"learning rate\", the multiplier of how much to modify neural network weights at each step, small value = small adjustments, big value = big adjustments).\n",
|
||
"* [`torch.optim.Adam(lr=0.001, params=model.parameters())`](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam) - The Adam optimizer (`params` stands for \"model parameters\", in other words, the model parameters/weights you'd like the optimization function to optimize during training)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 32,
|
||
"id": "e6ec2382-d11d-40d4-afb7-fec01f4abccb",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a baseline model\n",
|
||
"model = nn.Transformer()\n",
|
||
"\n",
|
||
"# SGD (stochastic gradient descent)\n",
|
||
"optimizer = torch.optim.SGD(lr=0.1, # set the learning rate (required)\n",
|
||
" params=model.parameters()) # tell the optimizer what parameters to optimize"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 33,
|
||
"id": "75a27e05-12fe-42f1-83aa-1e488f594373",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create a baseline model\n",
|
||
"model = nn.Transformer()\n",
|
||
"\n",
|
||
"# Adam optimizer\n",
|
||
"optimizer = torch.optim.Adam(lr=0.001, # set the learning rate (required)\n",
|
||
" params=model.parameters()) # tell the optimizer what parameters to optimize"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3e4bef22-4a74-487e-b8c6-18cb90bb0964",
|
||
"metadata": {},
|
||
"source": [
|
||
"## End-to-end example workflow\n",
|
||
"\n",
|
||
"Let's put everything together in a quick end-to-end workflow.\n",
|
||
"\n",
|
||
"<img src=\"https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/01_a_pytorch_workflow.png\" width=950 alt=\"a PyTorch workflow from data to building a model to fitting a model to evaluating the model\"/> \n",
|
||
"\n",
|
||
"This workflow has been taken from [01. PyTorch Workflow Fundamentals](https://www.learnpytorch.io/01_pytorch_workflow/)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "24805c73-7558-4d27-aba1-a78fce5312ce",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Create data"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 34,
|
||
"id": "7fed19d7-0e4e-42e8-ad80-8504033b3f6d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(tensor([[0.0000],\n",
|
||
" [0.0200],\n",
|
||
" [0.0400],\n",
|
||
" [0.0600],\n",
|
||
" [0.0800],\n",
|
||
" [0.1000],\n",
|
||
" [0.1200],\n",
|
||
" [0.1400],\n",
|
||
" [0.1600],\n",
|
||
" [0.1800]]),\n",
|
||
" tensor([[0.3000],\n",
|
||
" [0.3140],\n",
|
||
" [0.3280],\n",
|
||
" [0.3420],\n",
|
||
" [0.3560],\n",
|
||
" [0.3700],\n",
|
||
" [0.3840],\n",
|
||
" [0.3980],\n",
|
||
" [0.4120],\n",
|
||
" [0.4260]]))"
|
||
]
|
||
},
|
||
"execution_count": 34,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create *known* parameters\n",
|
||
"weight = 0.7\n",
|
||
"bias = 0.3\n",
|
||
"\n",
|
||
"# Create data\n",
|
||
"start = 0\n",
|
||
"end = 1\n",
|
||
"step = 0.02\n",
|
||
"X = torch.arange(start, end, step).unsqueeze(dim=1) # data\n",
|
||
"y = weight * X + bias # labels (want model to learn from data to predict these)\n",
|
||
"\n",
|
||
"X[:10], y[:10]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 35,
|
||
"id": "7dbfae76-edbf-470b-8686-8e12e30bbc8b",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(40, 40, 10, 10)"
|
||
]
|
||
},
|
||
"execution_count": 35,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"# Create train/test split\n",
|
||
"train_split = int(0.8 * len(X)) # 80% of data used for training set, 20% for testing \n",
|
||
"X_train, y_train = X[:train_split], y[:train_split]\n",
|
||
"X_test, y_test = X[train_split:], y[train_split:]\n",
|
||
"\n",
|
||
"len(X_train), len(y_train), len(X_test), len(y_test)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8b525ca1-bcc3-48c9-8b6b-ead556a50e9c",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Create a model\n",
|
||
"\n",
|
||
"Two main ways to create a model in PyTorch: \n",
|
||
"1. Subclass `torch.nn.Module` - more code but can be very flexible, models that subclass `torch.nn.Module` must implement a `forward()` method.\n",
|
||
"2. Use `torch.nn.Sequential` - less code but less flexibility."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 36,
|
||
"id": "0f102f8c-ad12-48b4-bfc6-bde5eb945bc8",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(LinearRegressionModel(\n",
|
||
" (linear_layer): Linear(in_features=1, out_features=1, bias=True)\n",
|
||
" ),\n",
|
||
" OrderedDict([('linear_layer.weight', tensor([[0.5025]])),\n",
|
||
" ('linear_layer.bias', tensor([-0.0722]))]))"
|
||
]
|
||
},
|
||
"execution_count": 36,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from torch import nn\n",
|
||
"\n",
|
||
"# Option 1 - subclass torch.nn.Module\n",
|
||
"class LinearRegressionModel(nn.Module):\n",
|
||
" def __init__(self):\n",
|
||
" super().__init__()\n",
|
||
" # Use nn.Linear() for creating the model parameters\n",
|
||
" self.linear_layer = nn.Linear(in_features=1, \n",
|
||
" out_features=1)\n",
|
||
" \n",
|
||
" # Define the forward computation (input data x flows through nn.Linear())\n",
|
||
" def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
|
||
" return self.linear_layer(x)\n",
|
||
"\n",
|
||
"model_0 = LinearRegressionModel()\n",
|
||
"model_0, model_0.state_dict()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "eb37ecd4-6258-404d-8dc3-149cd8540af3",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now let's create the same model as above but using `torch.nn.Sequential`."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 37,
|
||
"id": "e2c357fd-68ef-428e-90ae-ed225dad2e4d",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"text/plain": [
|
||
"(Sequential(\n",
|
||
" (0): Linear(in_features=1, out_features=1, bias=True)\n",
|
||
" ),\n",
|
||
" OrderedDict([('0.weight', tensor([[0.9905]])), ('0.bias', tensor([0.9053]))]))"
|
||
]
|
||
},
|
||
"execution_count": 37,
|
||
"metadata": {},
|
||
"output_type": "execute_result"
|
||
}
|
||
],
|
||
"source": [
|
||
"from torch import nn\n",
|
||
"\n",
|
||
"# Option 2 - use torch.nn.Sequential\n",
|
||
"model_1 = torch.nn.Sequential(\n",
|
||
" nn.Linear(in_features=1,\n",
|
||
" out_features=1))\n",
|
||
"\n",
|
||
"model_1, model_1.state_dict()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "11692983-6190-4d51-9d4e-5293f4205988",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Setup loss function and optimizer"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 38,
|
||
"id": "5c3d1c6d-a60f-4642-994d-7e0cb822164b",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"# Create loss function\n",
|
||
"loss_fn = nn.L1Loss()\n",
|
||
"\n",
|
||
"# Create optimizer\n",
|
||
"optimizer = torch.optim.SGD(params=model_1.parameters(), # optimize newly created model's parameters\n",
|
||
" lr=0.01)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d0b2f55e-8037-4942-8b74-0a09a3f4b007",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Create a training/testing loop\n",
|
||
"\n",
|
||
"Our goal is to reduce the loss of our model (how much our model's predictions are different to the actual data).\n",
|
||
"\n",
|
||
"If our training/testing loops are implemented right and the model is capable of learning patterns in the data, the training and test losses should go down.\n",
|
||
"\n",
|
||
"See the following for steps in a PyTorch training loop:\n",
|
||
"* [PyTorch optimization loop song](https://youtu.be/Nutpusq_AFw)\n",
|
||
"* [PyTorch Workflow Fundamentals Section 3: Training Loop](https://www.learnpytorch.io/01_pytorch_workflow/#pytorch-training-loop)\n",
|
||
"* [PyTorch Workflow Fundamentals Section 3: Testing Loop](https://www.learnpytorch.io/01_pytorch_workflow/#pytorch-testing-loop)\n",
|
||
"* [PyTorch Workflow Fundamentals Section 6.3: Training](https://www.learnpytorch.io/01_pytorch_workflow/#63-training)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 40,
|
||
"id": "0064882b-5fc7-4479-ad81-0dc0686ea711",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Epoch: 0 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 100 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 200 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 300 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 400 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 500 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 600 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 700 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 800 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n",
|
||
"Epoch: 900 | Train loss: 0.008362661115825176 | Test loss: 0.005596190690994263\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"torch.manual_seed(42)\n",
|
||
"\n",
|
||
"# Set the number of epochs \n",
|
||
"epochs = 1000 \n",
|
||
"\n",
|
||
"# Put data on the available device\n",
|
||
"# Without this, an error will happen (not all data on target device)\n",
|
||
"X_train = X_train.to(device)\n",
|
||
"X_test = X_test.to(device)\n",
|
||
"y_train = y_train.to(device)\n",
|
||
"y_test = y_test.to(device)\n",
|
||
"\n",
|
||
"# Put model on the available device\n",
|
||
"# With this, an error will happen (the model is not on target device)\n",
|
||
"model_1 = model_1.to(device)\n",
|
||
"\n",
|
||
"for epoch in range(epochs):\n",
|
||
" ### Training\n",
|
||
" model_1.train() # train mode is on by default after construction\n",
|
||
"\n",
|
||
" # 1. Forward pass\n",
|
||
" y_pred = model_1(X_train)\n",
|
||
"\n",
|
||
" # 2. Calculate loss\n",
|
||
" loss = loss_fn(y_pred, y_train)\n",
|
||
"\n",
|
||
" # 3. Zero grad optimizer\n",
|
||
" optimizer.zero_grad()\n",
|
||
"\n",
|
||
" # 4. Loss backward\n",
|
||
" loss.backward()\n",
|
||
"\n",
|
||
" # 5. Step the optimizer\n",
|
||
" optimizer.step()\n",
|
||
"\n",
|
||
" ### Testing\n",
|
||
" model_1.eval() # put the model in evaluation mode for testing (inference)\n",
|
||
" # 1. Forward pass\n",
|
||
" with torch.inference_mode():\n",
|
||
" test_pred = model_1(X_test)\n",
|
||
" \n",
|
||
" # 2. Calculate the loss\n",
|
||
" test_loss = loss_fn(test_pred, y_test)\n",
|
||
"\n",
|
||
" if epoch % 100 == 0:\n",
|
||
" print(f\"Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "af70942c-b2de-47ae-a266-d8097a124600",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Extras\n",
|
||
"\n",
|
||
"The above list is not exhaustive.\n",
|
||
"\n",
|
||
"Here are some good places to find out more:\n",
|
||
"\n",
|
||
"* [PyTorch official cheatsheet](https://pytorch.org/tutorials/beginner/ptcheat.html).\n",
|
||
"* [Zero to Mastery Learn PyTorch course](https://dbourke.link/ZTMPyTorch) - a comprehensive yet beginner-friendly deep dive into using PyTorch for deep learning all the way from the fundamentals to deploying a model to the real-world so other people can use it.\n",
|
||
"* [PyTorch performance tuning guide](https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html) - a resource from the PyTorch team on how to tune performance of PyTorch models.\n",
|
||
"* [PyTorch Extra Resources](https://www.learnpytorch.io/pytorch_extra_resources/) - a curated list of helpful resources to extend PyTorch and learn more about the engineering side of things around deep learning.\n",
|
||
"* [Effective PyTorch by vahidk](https://github.com/vahidk/EffectivePyTorch) - a GitHub repo with a fantastic overview of some of the main functionality in PyTorch in a straight-forward manner."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.0.0"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|