wehub/mrdbourke--pytorch-deep-learning
1
0
Fork 0
Code Issues Pull Requests Actions 1 Deployments Models Agent Notes Packages Projects Releases Wiki Activity
Files
main
mrdbourke--pytorch-deep-lea…/video_notebooks/00_pytorch_fundamentals_video.ipynb
T
wehub-resource-sync 37c25cd088
make_docs / deploy (push) Waiting to run
Details
chore: import upstream snapshot with attribution
2026-07-13 13:27:58 +08:00

2999 lines
79 KiB
Plaintext
Raw Permalink Blame History

{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "00_pytorch_fundamentals_video.ipynb",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true,
"authorship_tag": "ABX9TyMHZOb/mEFxyxNc+HBmyfpJ",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/video_notebooks/00_pytorch_fundamentals_video.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"## 00. PyTorch Fundamentals\n",
"\n",
"Resource notebook: https://www.learnpytorch.io/00_pytorch_fundamentals/\n",
"\n",
"If you have a question: https://github.com/mrdbourke/pytorch-deep-learning/discussions"
],
"metadata": {
"id": "3Wh_8cvFcDK9"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "37E0a1dNbfa4",
"outputId": "e94ca35e-c228-4334-c1ac-92cd9939ac68"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"1.10.0+cu111\n"
]
}
],
"source": [
"import torch\n",
"import pandas as pd\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"print(torch.__version__)"
]
},
{
"cell_type": "markdown",
"source": [
"## Introduction to Tensors\n",
"\n",
"### Creating tensors\n",
"\n",
"PyTorch tensors are created using `torch.Tensor()` = https://pytorch.org/docs/stable/tensors.html"
],
"metadata": {
"id": "JvYzOsXRdAVv"
}
},
{
"cell_type": "code",
"source": [
"# scalar\n",
"scalar = torch.tensor(7)\n",
"scalar"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "UTWI01UEdvni",
"outputId": "8ef0f4a9-7cd4-40e2-e7b9-cfefeb9f221f"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(7)"
]
},
"metadata": {},
"execution_count": 3
}
]
},
{
"cell_type": "code",
"source": [
"scalar.ndim"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5OQ5Mrxgd80W",
"outputId": "dc58e0fd-04a7-4d9c-8b88-14df0436a8a0"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0"
]
},
"metadata": {},
"execution_count": 4
}
]
},
{
"cell_type": "code",
"source": [
"# Get tensor back as Python int\n",
"scalar.item()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "CUDzqK6FehIE",
"outputId": "7c609b44-1e24-4629-e8bb-6711f814f27e"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"7"
]
},
"metadata": {},
"execution_count": 5
}
]
},
{
"cell_type": "code",
"source": [
"# Vector\n",
"vector = torch.tensor([7, 7])\n",
"vector"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LV0-tARqelWv",
"outputId": "6ad075f9-fad5-4d96-db28-e0b878334ffb"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([7, 7])"
]
},
"metadata": {},
"execution_count": 6
}
]
},
{
"cell_type": "code",
"source": [
"vector.ndim"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "whsizerteuKT",
"outputId": "6c09eaf9-730a-4dce-e36d-48935a02dad3"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1"
]
},
"metadata": {},
"execution_count": 7
}
]
},
{
"cell_type": "code",
"source": [
"vector.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FV6aI5Cpe6VT",
"outputId": "271b0a0a-05b0-410c-de4c-1356f8640a31"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.Size([2])"
]
},
"metadata": {},
"execution_count": 8
}
]
},
{
"cell_type": "code",
"source": [
"# MATRIX\n",
"MATRIX = torch.tensor([[7, 8],\n",
" [9, 10]])\n",
"MATRIX"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "NOTLQBwye_wo",
"outputId": "afdc9fb9-a8cd-4d21-de3f-5473d7fb952b"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[ 7, 8],\n",
" [ 9, 10]])"
]
},
"metadata": {},
"execution_count": 9
}
]
},
{
"cell_type": "code",
"source": [
"MATRIX.ndim"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3ArSidOCfPc_",
"outputId": "350b3e30-f07f-4207-c49a-a09e099101a5"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"2"
]
},
"metadata": {},
"execution_count": 10
}
]
},
{
"cell_type": "code",
"source": [
"MATRIX[1]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "BiIdkyk6fWhc",
"outputId": "2d7d2c70-13b4-4da9-9720-e20db452389c"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([ 9, 10])"
]
},
"metadata": {},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"source": [
"MATRIX.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QW1d-fKDfZHu",
"outputId": "c0a7b2ba-6d21-41c3-b85f-85fc57ef8fb9"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.Size([2, 2])"
]
},
"metadata": {},
"execution_count": 12
}
]
},
{
"cell_type": "code",
"source": [
"# TENSOR\n",
"TENSOR = torch.tensor([[[1, 2, 3],\n",
" [3, 6, 9],\n",
" [2, 4, 5]]])\n",
"\n",
"TENSOR"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GxOyqq8XffEx",
"outputId": "a2e4ae17-c902-4656-d8fd-63deb9a219e6"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[[1, 2, 3],\n",
" [3, 6, 9],\n",
" [2, 4, 5]]])"
]
},
"metadata": {},
"execution_count": 13
}
]
},
{
"cell_type": "code",
"source": [
"TENSOR.ndim"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "OwTwjbGDf0nk",
"outputId": "217df203-b0da-4349-d22f-93705e892ba6"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"3"
]
},
"metadata": {},
"execution_count": 14
}
]
},
{
"cell_type": "code",
"source": [
"TENSOR.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "xjffe97If9Q5",
"outputId": "1e49105e-8024-41dd-e9ec-3a0680f00777"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.Size([1, 3, 3])"
]
},
"metadata": {},
"execution_count": 15
}
]
},
{
"cell_type": "code",
"source": [
"TENSOR[0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "6NhtzkNygCGl",
"outputId": "c2e46751-0a3c-4925-9156-dbfe23290286"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[1, 2, 3],\n",
" [3, 6, 9],\n",
" [2, 4, 5]])"
]
},
"metadata": {},
"execution_count": 16
}
]
},
{
"cell_type": "markdown",
"source": [
"### Random tensors\n",
"\n",
"Why random tensors?\n",
"\n",
"Random tensors are important because the way many neural networks learn is that they start with tensors full of random numbers and then adjust those random numbers to better represent the data.\n",
"\n",
"`Start with random numbers -> look at data -> update random numbers -> look at data -> update random numbers`\n",
"\n",
"Torch random tensors - https://pytorch.org/docs/stable/generated/torch.rand.html"
],
"metadata": {
"id": "Sngk_9IXhUrI"
}
},
{
"cell_type": "code",
"source": [
"# Create a random tensor of size (3, 4)\n",
"random_tensor = torch.rand(3, 4)\n",
"random_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XC_LUjQQhsZN",
"outputId": "68bfbdbb-7e7a-461d-fbab-0054b16cd3ac"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[0.4433, 0.7119, 0.4170, 0.4409],\n",
" [0.8014, 0.2050, 0.3547, 0.6358],\n",
" [0.3007, 0.1659, 0.3462, 0.7317]])"
]
},
"metadata": {},
"execution_count": 17
}
]
},
{
"cell_type": "code",
"source": [
"# Create a random tensor with similar shape to an image tensor\n",
"random_image_size_tensor = torch.rand(size=(3, 224, 224)) # height, width, colour channels (R, G, B)\n",
"random_image_size_tensor.shape, random_image_size_tensor.ndim"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HNb5EHcyh_9v",
"outputId": "229c6c05-867b-4bbb-9c8e-5c03554c3850"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(torch.Size([3, 224, 224]), 3)"
]
},
"metadata": {},
"execution_count": 18
}
]
},
{
"cell_type": "markdown",
"source": [
"### Zeros and ones"
],
"metadata": {
"id": "U2FL8Tqkh_6s"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor of all zeros\n",
"zeros = torch.zeros(size=(3, 4))\n",
"zeros"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "PXLbKXX7h_4X",
"outputId": "41735321-efc6-465f-ff00-7907f19c2640"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[0., 0., 0., 0.],\n",
" [0., 0., 0., 0.],\n",
" [0., 0., 0., 0.]])"
]
},
"metadata": {},
"execution_count": 19
}
]
},
{
"cell_type": "code",
"source": [
"# Create a tensor of all ones\n",
"ones = torch.ones(size=(3, 4))\n",
"ones"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "zvsnxHclh_1T",
"outputId": "735cb920-e2bb-423d-bb8a-7a2812623d68"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[1., 1., 1., 1.],\n",
" [1., 1., 1., 1.],\n",
" [1., 1., 1., 1.]])"
]
},
"metadata": {},
"execution_count": 20
}
]
},
{
"cell_type": "code",
"source": [
"ones.dtype"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HgZwd5SLh_yQ",
"outputId": "54785d1f-2d65-4108-99dc-dc7c87497c4e"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.float32"
]
},
"metadata": {},
"execution_count": 21
}
]
},
{
"cell_type": "code",
"source": [
"random_tensor.dtype"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "O58wXELFh_vr",
"outputId": "eb3b1a05-e281-4857-8887-3ab4b049f80b"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.float32"
]
},
"metadata": {},
"execution_count": 22
}
]
},
{
"cell_type": "markdown",
"source": [
"### Creating a range of tensors and tensors-like "
],
"metadata": {
"id": "A9oyiP_yh_tG"
}
},
{
"cell_type": "code",
"source": [
"# Use torch.range() and get deprecated message, use torch.arange()\n",
"one_to_ten = torch.arange(start=1, end=11, step=1)\n",
"one_to_ten"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MXv_6ZRzkg48",
"outputId": "c33aa934-d620-4be9-e4b4-b8414802e92b"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])"
]
},
"metadata": {},
"execution_count": 23
}
]
},
{
"cell_type": "code",
"source": [
"# Creating tensors like\n",
"ten_zeros = torch.zeros_like(input=one_to_ten)\n",
"ten_zeros"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8jnWbY-ikhgd",
"outputId": "d4bb2850-51b7-46a9-e4d0-39c5e5f98163"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
]
},
"metadata": {},
"execution_count": 24
}
]
},
{
"cell_type": "markdown",
"source": [
"### Tensor datatypes \n",
"\n",
"**Note:** Tensor datatypes is one of the 3 big errors you'll run into with PyTorch & deep learning:\n",
"1. Tensors not right datatype\n",
"2. Tensors not right shape\n",
"3. Tensors not on the right device\n",
"\n",
"Precision in computing - https://en.wikipedia.org/wiki/Precision_(computer_science)#:~:text=In%20computer%20science%2C%20the%20precision,used%20to%20express%20a%20value."
],
"metadata": {
"id": "gZAIvQyzm2E2"
}
},
{
"cell_type": "code",
"source": [
"# Float 32 tensor\n",
"float_32_tensor = torch.tensor([3.0, 6.0, 9.0],\n",
" dtype=None, # what datatype is the tensor (e.g. float32 or float16)\n",
" device=None, # What device is your tensor on\n",
" requires_grad=False) # whether or not to track gradients with this tensors operations\n",
"float_32_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "TEyHrEWFnO78",
"outputId": "1fcbfb44-82b3-4eb6-dd15-473d5235acd7"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([3., 6., 9.])"
]
},
"metadata": {},
"execution_count": 25
}
]
},
{
"cell_type": "code",
"source": [
"float_32_tensor.dtype"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wke5_RItnWxS",
"outputId": "89660a90-feba-4423-d79d-7885bc66cfb4"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"torch.float32"
]
},
"metadata": {},
"execution_count": 26
}
]
},
{
"cell_type": "code",
"source": [
"float_16_tensor = float_32_tensor.type(torch.float16)\n",
"float_16_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vqKcjujanY2J",
"outputId": "45f42da5-4f86-4850-8337-58aff436a9cb"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([3., 6., 9.], dtype=torch.float16)"
]
},
"metadata": {},
"execution_count": 27
}
]
},
{
"cell_type": "code",
"source": [
"float_16_tensor * float_32_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JBerGK0MpkFA",
"outputId": "a856bc6a-2507-45da-9976-b87e86f54069"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([ 9., 36., 81.])"
]
},
"metadata": {},
"execution_count": 28
}
]
},
{
"cell_type": "code",
"source": [
"int_32_tensor = torch.tensor([3, 6, 9], dtype=torch.long)\n",
"int_32_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "NpgQSORsp9J6",
"outputId": "a3b257ce-6215-4f43-838a-55dfd1807f65"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([3, 6, 9])"
]
},
"metadata": {},
"execution_count": 29
}
]
},
{
"cell_type": "code",
"source": [
"float_32_tensor * int_32_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9A6eAgP1qQLO",
"outputId": "bcb9497b-42d4-47cd-ffea-cd2e987bd09e"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([ 9., 36., 81.])"
]
},
"metadata": {},
"execution_count": 30
}
]
},
{
"cell_type": "markdown",
"source": [
"### Getting information from tensors (tensor attributes)\n",
"\n",
"1. Tensors not right datatype - to do get datatype from a tensor, can use `tensor.dtype`\n",
"2. Tensors not right shape - to get shape from a tensor, can use `tensor.shape`\n",
"3. Tensors not on the right device - to get device from a tensor, can use `tensor.device`"
],
"metadata": {
"id": "g-JhM8bzqUTP"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor\n",
"some_tensor = torch.rand(3, 4)\n",
"some_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MTmy1K8NrA7q",
"outputId": "ebce8bc4-9e1d-4040-df96-6343ba3c9354"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[0.7151, 0.9288, 0.0464, 0.2910],\n",
" [0.7281, 0.5272, 0.9098, 0.3145],\n",
" [0.9641, 0.4652, 0.8553, 0.0232]])"
]
},
"metadata": {},
"execution_count": 31
}
]
},
{
"cell_type": "code",
"source": [
"# Find out details about some tensor\n",
"print(some_tensor)\n",
"print(f\"Datatype of tensor: {some_tensor.dtype}\")\n",
"print(f\"Shape of tensor: {some_tensor.shape}\")\n",
"print(f\"Device tensor is on: {some_tensor.device}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "EVdSeMN9rG3t",
"outputId": "02cb4a54-1699-49b2-d5be-b095aaac92ef"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor([[0.7151, 0.9288, 0.0464, 0.2910],\n",
" [0.7281, 0.5272, 0.9098, 0.3145],\n",
" [0.9641, 0.4652, 0.8553, 0.0232]])\n",
"Datatype of tensor: torch.float32\n",
"Shape of tensor: torch.Size([3, 4])\n",
"Device tensor is on: cpu\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"### Manipulating Tensors (tensor operations)\n",
"\n",
"Tensor opertions include:\n",
"* Addition\n",
"* Subtraction\n",
"* Multiplication (element-wise)\n",
"* Division\n",
"* Matrix multiplication"
],
"metadata": {
"id": "vGncmTNOrsEN"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor and add 10 to it\n",
"tensor = torch.tensor([1, 2, 3])\n",
"tensor + 10"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1UEudneS6Opn",
"outputId": "92387bfa-caed-4525-8c64-7b07fdeee104"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([11, 12, 13])"
]
},
"metadata": {},
"execution_count": 33
}
]
},
{
"cell_type": "code",
"source": [
"# Multiply tensor by 10\n",
"tensor * 10"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7_a_8nnJ6Xsy",
"outputId": "170fc190-01f9-41a3-c8b3-6509e2c23cca"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([10, 20, 30])"
]
},
"metadata": {},
"execution_count": 34
}
]
},
{
"cell_type": "code",
"source": [
"tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "363dnmSi6fg5",
"outputId": "c7dc4a2d-210f-431a-d2fd-ce779312ba64"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3])"
]
},
"metadata": {},
"execution_count": 35
}
]
},
{
"cell_type": "code",
"source": [
"# Substract 10\n",
"tensor - 10"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Ab5Ijquc6gn5",
"outputId": "3855276a-a447-40c8-ddeb-8b836b531e76"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([-9, -8, -7])"
]
},
"metadata": {},
"execution_count": 36
}
]
},
{
"cell_type": "code",
"source": [
"# Try out PyTorch in-built functions\n",
"torch.mul(tensor, 10)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KlwwWMwQ6s7Z",
"outputId": "805f1fdc-1208-4a3f-8615-9c87eeeea8ca"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([10, 20, 30])"
]
},
"metadata": {},
"execution_count": 37
}
]
},
{
"cell_type": "code",
"source": [
"torch.add(tensor, 10)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KxV6flg_604F",
"outputId": "8513f4c4-68ac-42cc-ecbd-0f51f40052f4"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([11, 12, 13])"
]
},
"metadata": {},
"execution_count": 38
}
]
},
{
"cell_type": "markdown",
"source": [
"### Matrix multiplication\n",
"\n",
"Two main ways of performing multiplication in neural networks and deep learning:\n",
"\n",
"1. Element-wise multiplication\n",
"2. Matrix mutliplication (dot product)\n",
"\n",
"More information on multiplying matrices - https://www.mathsisfun.com/algebra/matrix-multiplying.html\n",
"\n",
"There are two main rules that performing matrix mutliplication needs to satisfy:\n",
"1. The **inner dimensions** must match:\n",
"* `(3, 2) @ (3, 2)` won't work\n",
"* `(2, 3) @ (3, 2)` will work\n",
"* `(3, 2) @ (2, 3)` will work\n",
"2. The resulting matrix has the shape of the **outer dimensions**:\n",
"* `(2, 3) @ (3, 2)` -> `(2, 2)`\n",
"* `(3, 2) @ (2, 3)` -> `(3, 3)`"
],
"metadata": {
"id": "2HLSVYm6-bXw"
}
},
{
"cell_type": "code",
"source": [
"# Element wise multiplication\n",
"print(tensor, \"*\", tensor)\n",
"print(f\"Equals: {tensor * tensor}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "AL0Ef09oAPwz",
"outputId": "8ffd729e-7b8f-4b25-934a-3485a88fef5f"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor([1, 2, 3]) * tensor([1, 2, 3])\n",
"Equals: tensor([1, 4, 9])\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Matrix multiplication\n",
"torch.matmul(tensor, tensor)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "MVinO8q8AVgu",
"outputId": "67961e52-a953-41e7-9bab-823521bf31a0"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(14)"
]
},
"metadata": {},
"execution_count": 40
}
]
},
{
"cell_type": "code",
"source": [
"tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JQqozJdQA1dP",
"outputId": "2174de30-0909-41bc-ee0d-f09ee534ea69"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3])"
]
},
"metadata": {},
"execution_count": 41
}
]
},
{
"cell_type": "code",
"source": [
"# Matrix multiplication by hand\n",
"1*1 + 2*2 + 3*3"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "euQ9rTuvAl5y",
"outputId": "a87fc8d2-96a5-4135-99c5-9f165ea3e971"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"14"
]
},
"metadata": {},
"execution_count": 42
}
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"value = 0\n",
"for i in range(len(tensor)):\n",
" value += tensor[i] * tensor[i]\n",
"print(value)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ay6a2Ti1A9mc",
"outputId": "cdf67c77-09f2-4269-afa4-68e617609a84"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor(14)\n",
"CPU times: user 661 µs, sys: 876 µs, total: 1.54 ms\n",
"Wall time: 1.55 ms\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"torch.matmul(tensor, tensor)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "jBp_AC3YBTXX",
"outputId": "bab5b7cb-acc2-49b1-b8fd-3a545e0148da"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 67 µs, sys: 29 µs, total: 96 µs\n",
"Wall time: 101 µs\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(14)"
]
},
"metadata": {},
"execution_count": 44
}
]
},
{
"cell_type": "markdown",
"source": [
"### One of the most common errors in deep learning: shape errors"
],
"metadata": {
"id": "VmX8MOCoBZMu"
}
},
{
"cell_type": "code",
"source": [
"# Shapes for matrix multiplication \n",
"tensor_A = torch.tensor([[1, 2],\n",
" [3, 4],\n",
" [5, 6]])\n",
"\n",
"tensor_B = torch.tensor([[7, 10],\n",
" [8, 11],\n",
" [9, 12]])\n",
"\n",
"# torch.mm(tensor_A, tensor_B) # torch.mm is the same as torch.matmul (it's an alias for writing less code)\n",
"torch.matmul(tensor_A, tensor_B)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 203
},
"id": "M2rpP-bxCmef",
"outputId": "2ded8074-f6a0-465a-e6eb-720a5a70dec2"
},
"execution_count": null,
"outputs": [
{
"output_type": "error",
"ename": "RuntimeError",
"evalue": "ignored",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-46-281a2d72c2ec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;31m# torch.mm(tensor_A, tensor_B) # torch.mm is the same as torch.matmul (it's an alias for writing less code)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor_A\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor_B\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m: mat1 and mat2 shapes cannot be multiplied (3x2 and 3x2)"
]
}
]
},
{
"cell_type": "code",
"source": [
"tensor_B.T"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Zzrun--YHgzV",
"outputId": "e73a749b-508b-4124-a3a2-2e41a840e4df"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[ 7, 8, 9],\n",
" [10, 11, 12]])"
]
},
"metadata": {},
"execution_count": 47
}
]
},
{
"cell_type": "code",
"source": [
"tensor_A.shape, tensor_B.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Iskru-L3FQyy",
"outputId": "f2f2c310-b5ff-4835-8d09-93b5005d82e4"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(torch.Size([3, 2]), torch.Size([3, 2]))"
]
},
"metadata": {},
"execution_count": 48
}
]
},
{
"cell_type": "markdown",
"source": [
"To fix our tensor shape issues, we can manipulate the shape of one of our tensors using a **transpose**.\n",
"\n",
"A **transpose** switches the axes or dimensions of a given tensor."
],
"metadata": {
"id": "qd5aRZjgFUlV"
}
},
{
"cell_type": "code",
"source": [
"tensor_B, tensor_B.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gaPS2iZSFnkY",
"outputId": "67f9f7fe-4877-4cf8-8bd5-8065bf686dc4"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[ 7, 10],\n",
" [ 8, 11],\n",
" [ 9, 12]]), torch.Size([3, 2]))"
]
},
"metadata": {},
"execution_count": 49
}
]
},
{
"cell_type": "code",
"source": [
"tensor_B.T, tensor_B.T.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "8HPpvbCSFkWs",
"outputId": "e351a3d7-3e17-416f-ed93-80ffd30cdfa8"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[ 7, 8, 9],\n",
" [10, 11, 12]]), torch.Size([2, 3]))"
]
},
"metadata": {},
"execution_count": 50
}
]
},
{
"cell_type": "code",
"source": [
"# The matrix multiplication operation works when tensor_B is transposed\n",
"print(f\"Original shapes: tensor_A = {tensor_A.shape}, tensor_B = {tensor_B.shape}\")\n",
"print(f\"New shapes: tensor_A = {tensor_A.shape} (same shape as above), tensor_B.T = {tensor_B.T.shape}\")\n",
"print(f\"Multiplying: {tensor_A.shape} @ {tensor_B.T.shape} <- inner dimensions must match\")\n",
"print(\"Output:\\n\")\n",
"output = torch.matmul(tensor_A, tensor_B.T)\n",
"print(output) \n",
"print(f\"\\nOutput shape: {output.shape}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "yIcUJL_DFqrs",
"outputId": "1c7f4db8-5ac0-49c1-d983-e93fb1b5bf0c"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Original shapes: tensor_A = torch.Size([3, 2]), tensor_B = torch.Size([3, 2])\n",
"New shapes: tensor_A = torch.Size([3, 2]) (same shape as above), tensor_B.T = torch.Size([2, 3])\n",
"Multiplying: torch.Size([3, 2]) @ torch.Size([2, 3]) <- inner dimensions must match\n",
"Output:\n",
"\n",
"tensor([[ 27, 30, 33],\n",
" [ 61, 68, 75],\n",
" [ 95, 106, 117]])\n",
"\n",
"Output shape: torch.Size([3, 3])\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## Finding the min, max, mean, sum, etc (tensor aggregation)"
],
"metadata": {
"id": "-2Q9lQ10F-sx"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor\n",
"x = torch.arange(1, 100, 10)\n",
"x, x.dtype"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vmGm37iiIx_6",
"outputId": "f662f4e9-5524-4871-8c2c-35aa30e61636"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([ 1, 11, 21, 31, 41, 51, 61, 71, 81, 91]), torch.int64)"
]
},
"metadata": {},
"execution_count": 52
}
]
},
{
"cell_type": "code",
"source": [
"# Find the min\n",
"torch.min(x), x.min()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "l7KbfOFeI8Wn",
"outputId": "10715772-4bcd-40c4-ef9a-a004800f6bf8"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor(1), tensor(1))"
]
},
"metadata": {},
"execution_count": 53
}
]
},
{
"cell_type": "code",
"source": [
"# Find the max\n",
"torch.max(x), x.max()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "eQk93EJaI-j8",
"outputId": "fff58ef6-be2e-40d0-8050-6de4393d3e18"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor(91), tensor(91))"
]
},
"metadata": {},
"execution_count": 54
}
]
},
{
"cell_type": "code",
"source": [
"# Find the mean - note: the torch.mean() function requires a tensor of float32 datatype to work\n",
"torch.mean(x.type(torch.float32)), x.type(torch.float32).mean()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "YdNwHqMBJC-n",
"outputId": "b2d38991-b734-468f-c79a-c01fc0b6254d"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor(46.), tensor(46.))"
]
},
"metadata": {},
"execution_count": 55
}
]
},
{
"cell_type": "code",
"source": [
"# Find the sum\n",
"torch.sum(x), x.sum()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0N9ppySsJHVY",
"outputId": "6b8603f0-f16f-4b79-9379-64c1e20273ee"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor(460), tensor(460))"
]
},
"metadata": {},
"execution_count": 56
}
]
},
{
"cell_type": "markdown",
"source": [
"## Finding the positional min and max"
],
"metadata": {
"id": "kLa91TZiJ31t"
}
},
{
"cell_type": "code",
"source": [
"x"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "eIG9GRbyKt0t",
"outputId": "7be0fa2e-b125-4b11-a77b-7fd7aae2a4bf"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([ 1, 11, 21, 31, 41, 51, 61, 71, 81, 91])"
]
},
"metadata": {},
"execution_count": 57
}
]
},
{
"cell_type": "code",
"source": [
"# Find the position in tensor that has the minimum value with argmin() -> returns index position of targt tensor where the minimum value occurs \n",
"x.argmin()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "LfYi96igKsQT",
"outputId": "10ef9563-7789-4d8e-9bd1-a6091e5c27f4"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(0)"
]
},
"metadata": {},
"execution_count": 58
}
]
},
{
"cell_type": "code",
"source": [
"x[0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Jc5GKGO6KtWp",
"outputId": "89a9dfaf-581d-484d-e42a-b6de95584256"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(1)"
]
},
"metadata": {},
"execution_count": 59
}
]
},
{
"cell_type": "code",
"source": [
"# Find the position in tensor that has the maximum value with argmax()\n",
"x.argmax()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FBGnJblVKxu1",
"outputId": "77334a4f-ffd7-4ba4-e700-fe21750102b0"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(9)"
]
},
"metadata": {},
"execution_count": 60
}
]
},
{
"cell_type": "code",
"source": [
"x[9]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "X9QQXAKaLEJ_",
"outputId": "b30e9115-23d7-4429-ac9a-e6119060c966"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(91)"
]
},
"metadata": {},
"execution_count": 61
}
]
},
{
"cell_type": "markdown",
"source": [
"## Reshaping, stacking, squeezing and unsqueezing tensors\n",
"\n",
"* Reshaping - reshapes an input tensor to a defined shape\n",
"* View - Return a view of an input tensor of certain shape but keep the same memory as the original tensor\n",
"* Stacking - combine multiple tensors on top of each other (vstack) or side by side (hstack)\n",
"* Squeeze - removes all `1` dimensions from a tensor\n",
"* Unsqueeze - add a `1` dimension to a target tensor\n",
"* Permute - Return a view of the input with dimensions permuted (swapped) in a certain way"
],
"metadata": {
"id": "ZPcyfriKLHXT"
}
},
{
"cell_type": "code",
"source": [
"# Let's create a tensor\n",
"import torch\n",
"x = torch.arange(1., 10.)\n",
"x, x.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "vRJedWy3LLjK",
"outputId": "d616e6f4-2d72-41e9-b165-e277335b3cca"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([1., 2., 3., 4., 5., 6., 7., 8., 9.]), torch.Size([9]))"
]
},
"metadata": {},
"execution_count": 77
}
]
},
{
"cell_type": "code",
"source": [
"# Add an extra dimension\n",
"x_reshaped = x.reshape(1, 9)\n",
"x_reshaped, x_reshaped.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9scaX6wcMcv_",
"outputId": "4037fb10-0443-4752-9ca0-9f91a9d48946"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9.]]), torch.Size([1, 9]))"
]
},
"metadata": {},
"execution_count": 78
}
]
},
{
"cell_type": "code",
"source": [
"# Change the view \n",
"z = x.view(1, 9)\n",
"z, z.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "-cQDwoKGMcqC",
"outputId": "53f6d9f7-1489-4b11-c7a7-6dd9f28ce7d7"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9.]]), torch.Size([1, 9]))"
]
},
"metadata": {},
"execution_count": 79
}
]
},
{
"cell_type": "code",
"source": [
"# Changing z changes x (because a view of a tensor shares the same memory as the original input)\n",
"z[:, 0] = 5\n",
"z, x"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "x4ZmA_h4McRP",
"outputId": "7fb6bdd8-077f-485a-ebde-91b2c36dd46f"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]]),\n",
" tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.]))"
]
},
"metadata": {},
"execution_count": 80
}
]
},
{
"cell_type": "code",
"source": [
"# Stack tensors on top of each other\n",
"x_stacked = torch.stack([x, x, x, x], dim=0)\n",
"x_stacked"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "u1Fi833cNudG",
"outputId": "f071a3d6-37bf-415c-b6ba-55c4655baede"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
" [5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
" [5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
" [5., 2., 3., 4., 5., 6., 7., 8., 9.]])"
]
},
"metadata": {},
"execution_count": 87
}
]
},
{
"cell_type": "code",
"source": [
"# torch.squeeze() - removes all single dimensions from a target tensor\n",
"print(f\"Previous tensor: {x_reshaped}\")\n",
"print(f\"Previous shape: {x_reshaped.shape}\")\n",
"\n",
"# Remove extra dimensions from x_reshaped\n",
"x_squeezed = x_reshaped.squeeze()\n",
"print(f\"\\nNew tensor: {x_squeezed}\")\n",
"print(f\"New shape: {x_squeezed.shape}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "F6XzyHuMOIJb",
"outputId": "9ffba5d6-ee6a-4c09-a052-fb084a1afc67"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Previous tensor: tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]])\n",
"Previous shape: torch.Size([1, 9])\n",
"\n",
"New tensor: tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.])\n",
"New shape: torch.Size([9])\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# torch.unsqueeze() - adds a single dimension to a target tensor at a specific dim (dimension)\n",
"print(f\"Previous target: {x_squeezed}\")\n",
"print(f\"Previous shape: {x_squeezed.shape}\")\n",
"\n",
"# Add an extra dimension with unsqueeze\n",
"x_unsqueezed = x_squeezed.unsqueeze(dim=0)\n",
"print(f\"\\nNew tensor: {x_unsqueezed}\")\n",
"print(f\"New shape: {x_unsqueezed.shape}\")"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "J9UTmuhvPGwV",
"outputId": "bb3f01dc-33a3-44f5-d042-2adcc4bb202b"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Previous target: tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.])\n",
"Previous shape: torch.Size([9])\n",
"\n",
"New tensor: tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]])\n",
"New shape: torch.Size([1, 9])\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# torch.permute - rearranges the dimensions of a target tensor in a specified order\n",
"x_original = torch.rand(size=(224, 224, 3)) # [height, width, colour_channels]\n",
"\n",
"# Permute the original tensor to rearrange the axis (or dim) order\n",
"x_permuted = x_original.permute(2, 0, 1) # shifts axis 0->1, 1->2, 2->0\n",
"\n",
"print(f\"Previous shape: {x_original.shape}\") \n",
"print(f\"New shape: {x_permuted.shape}\") # [colour_channels, height, width]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rBGYuy2TPHra",
"outputId": "bf9d9a76-0156-4a80-8538-f57a462a3b1d"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Previous shape: torch.Size([224, 224, 3])\n",
"New shape: torch.Size([3, 224, 224])\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"x_original[0, 0, 0] = 728218\n",
"x_original[0, 0, 0], x_permuted[0, 0, 0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "x5uR_Il-PQOF",
"outputId": "e7868c1f-9475-4740-c594-698a4a43a8cb"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor(728218.), tensor(728218.))"
]
},
"metadata": {},
"execution_count": 104
}
]
},
{
"cell_type": "markdown",
"source": [
"## Indexing (selecting data from tensors)\n",
"\n",
"Indexing with PyTorch is similar to indexing with NumPy."
],
"metadata": {
"id": "r8P5o0gdRh0x"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor\n",
"import torch\n",
"x = torch.arange(1, 10).reshape(1, 3, 3)\n",
"x, x.shape"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "okvAfJ3ASinI",
"outputId": "ec5155c1-cada-4e4f-d9f9-cbdb2ea361e3"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([[[1, 2, 3],\n",
" [4, 5, 6],\n",
" [7, 8, 9]]]), torch.Size([1, 3, 3]))"
]
},
"metadata": {},
"execution_count": 105
}
]
},
{
"cell_type": "code",
"source": [
"# Let's index on our new tensor\n",
"x[0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "5hU35njXSnPi",
"outputId": "10cdbc82-c7d1-49d3-f7cc-ab0fcec81ed7"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[1, 2, 3],\n",
" [4, 5, 6],\n",
" [7, 8, 9]])"
]
},
"metadata": {},
"execution_count": 106
}
]
},
{
"cell_type": "code",
"source": [
"# Let's index on the middle bracket (dim=1)\n",
"x[0][0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "pqFyG5S7SnKS",
"outputId": "ead28195-f9b2-4944-9308-1154ad2a2e75"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3])"
]
},
"metadata": {},
"execution_count": 109
}
]
},
{
"cell_type": "code",
"source": [
"# Let's index on the most inner bracket (last dimension)\n",
"x[0][1][1]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Pf6GfxwBSnDL",
"outputId": "bf6c471f-a5c9-4d32-83fb-6af71bd2bf5e"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor(5)"
]
},
"metadata": {},
"execution_count": 113
}
]
},
{
"cell_type": "code",
"source": [
"# You can also use \":\" to select \"all\" of a target dimension\n",
"x[:, 0]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "PQH4EttRSmyv",
"outputId": "32d699f8-36ba-4f3f-9cb3-758f27d1cb15"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[1, 2, 3]])"
]
},
"metadata": {},
"execution_count": 114
}
]
},
{
"cell_type": "code",
"source": [
"# Get all values of 0th and 1st dimensions but only index 1 of 2nd dimension\n",
"x[:, :, 1]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "cdZcxA7ITuyt",
"outputId": "94ba1d6b-2a40-44e5-bff1-fb75c6fe8b83"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([[2, 5, 8]])"
]
},
"metadata": {},
"execution_count": 115
}
]
},
{
"cell_type": "code",
"source": [
"# Get all values of the 0 dimension but only the 1 index value of 1st and 2nd dimension\n",
"x[:, 1, 1]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "hqv2hRz7T245",
"outputId": "1894d5f5-1c9b-4b75-8c17-97e77e709557"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([5])"
]
},
"metadata": {},
"execution_count": 118
}
]
},
{
"cell_type": "code",
"source": [
"# Get index 0 of 0th and 1st dimension and all values of 2nd dimension\n",
"x[0, 0, :]"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QSXRblctUGTw",
"outputId": "dc5ea85d-0d03-4774-b27f-a68ed219aa31"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3])"
]
},
"metadata": {},
"execution_count": 119
}
]
},
{
"cell_type": "code",
"source": [
"# Index on x to return 9\n",
"print(x[0][2][2])\n",
"\n",
"# Index on x to return 3, 6, 9\n",
"print(x[:, :, 2])"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "DOCotUAhUT0R",
"outputId": "caaeb65c-af20-4d2e-ecfc-1ec2923398ec"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor(9)\n",
"tensor([[3, 6, 9]])\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"## PyTorch tensors & NumPy\n",
"\n",
"NumPy is a popular scientific Python numerical computing library. \n",
"\n",
"And because of this, PyTorch has functionality to interact with it.\n",
"\n",
"* Data in NumPy, want in PyTorch tensor -> `torch.from_numpy(ndarray)`\n",
"* PyTorch tensor -> NumPy -> `torch.Tensor.numpy()` "
],
"metadata": {
"id": "1OjawP9XUcEr"
}
},
{
"cell_type": "code",
"source": [
"# NumPy array to tensor\n",
"import torch\n",
"import numpy as np\n",
"\n",
"array = np.arange(1.0, 8.0)\n",
"tensor = torch.from_numpy(array) # warning: when converting from numpy -> pytorch, pytorch reflects numpy's default datatype of float64 unless specified otherwise\n",
"array, tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "2I1qZb9dU150",
"outputId": "306cd520-ffe8-4915-b315-9e49ca67994f"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(array([1., 2., 3., 4., 5., 6., 7.]),\n",
" tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))"
]
},
"metadata": {},
"execution_count": 135
}
]
},
{
"cell_type": "code",
"source": [
"# Change the value of array, what will this do to `tensor`?\n",
"array = array + 1\n",
"array, tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "D2D2_7EgU104",
"outputId": "98fbe0f6-87ca-49c4-98c5-4ecb3a88fef1"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(array([2., 3., 4., 5., 6., 7., 8.]),\n",
" tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))"
]
},
"metadata": {},
"execution_count": 136
}
]
},
{
"cell_type": "code",
"source": [
"# Tensor to NumPy array\n",
"tensor = torch.ones(7)\n",
"numpy_tensor = tensor.numpy()\n",
"tensor, numpy_tensor"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "uVX9OHluU1v9",
"outputId": "b655993d-f0dd-4361-efb8-644a0f709262"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([1., 1., 1., 1., 1., 1., 1.]),\n",
" array([1., 1., 1., 1., 1., 1., 1.], dtype=float32))"
]
},
"metadata": {},
"execution_count": 137
}
]
},
{
"cell_type": "code",
"source": [
"# Change the tesnor, what happens to `numpy_tensor`?\n",
"tensor = tensor + 1\n",
"tensor, numpy_tensor "
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "olqJ6oFtU1hF",
"outputId": "17235a8b-fb54-4b2c-dc1d-55f4b6f77e46"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(tensor([2., 2., 2., 2., 2., 2., 2.]),\n",
" array([1., 1., 1., 1., 1., 1., 1.], dtype=float32))"
]
},
"metadata": {},
"execution_count": 139
}
]
},
{
"cell_type": "markdown",
"source": [
"## Reproducbility (trying to take random out of random)\n",
"\n",
"In short how a neural network learns:\n",
"\n",
"`start with random numbers -> tensor operations -> update random numbers to try and make them better representations of the data -> again -> again -> again...`\n",
"\n",
"To reduce the randomness in neural networks and PyTorch comes the concept of a **random seed**.\n",
"\n",
"Essentially what the random seed does is \"flavour\" the randomness."
],
"metadata": {
"id": "cWYQ6KRjW7cu"
}
},
{
"cell_type": "code",
"source": [
"import torch\n",
"\n",
"# Create two random tensors\n",
"random_tensor_A = torch.rand(3, 4)\n",
"random_tensor_B = torch.rand(3, 4)\n",
"\n",
"print(random_tensor_A)\n",
"print(random_tensor_B)\n",
"print(random_tensor_A == random_tensor_B)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "inL6FZxgYBYa",
"outputId": "c3fdab3b-fb6d-4476-d019-41a3246b965e"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor([[0.3675, 0.8410, 0.0507, 0.3165],\n",
" [0.7275, 0.9676, 0.3901, 0.8840],\n",
" [0.5177, 0.2239, 0.4362, 0.3602]])\n",
"tensor([[0.5229, 0.6719, 0.2790, 0.8198],\n",
" [0.6689, 0.8659, 0.7849, 0.4268],\n",
" [0.2076, 0.8076, 0.4377, 0.2555]])\n",
"tensor([[False, False, False, False],\n",
" [False, False, False, False],\n",
" [False, False, False, False]])\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Let's make some random but reproducible tensors\n",
"import torch\n",
"\n",
"# Set the random seed\n",
"RANDOM_SEED = 42\n",
"torch.manual_seed(RANDOM_SEED)\n",
"random_tensor_C = torch.rand(3, 4)\n",
"\n",
"torch.manual_seed(RANDOM_SEED)\n",
"random_tensor_D = torch.rand(3, 4)\n",
"\n",
"print(random_tensor_C)\n",
"print(random_tensor_D)\n",
"print(random_tensor_C == random_tensor_D)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "EFGlNq_0Yskj",
"outputId": "1697a05c-0a1d-4f9f-c704-52504955d173"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
" [0.3904, 0.6009, 0.2566, 0.7936],\n",
" [0.9408, 0.1332, 0.9346, 0.5936]])\n",
"tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
" [0.3904, 0.6009, 0.2566, 0.7936],\n",
" [0.9408, 0.1332, 0.9346, 0.5936]])\n",
"tensor([[True, True, True, True],\n",
" [True, True, True, True],\n",
" [True, True, True, True]])\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"Extra resources for reproducibility:\n",
"* https://pytorch.org/docs/stable/notes/randomness.html\n",
"* https://en.wikipedia.org/wiki/Random_seed"
],
"metadata": {
"id": "T8IDM0VTZdqA"
}
},
{
"cell_type": "markdown",
"source": [
" ## Running tensors and PyTorch objects on the GPUs (and making faster computations)\n",
"\n",
" GPUs = faster computation on numbers, thanks to CUDA + NVIDIA hardware + PyTorch working behind the scenes to make everything hunky dory (good)."
],
"metadata": {
"id": "mw3pNVM9aSTb"
}
},
{
"cell_type": "markdown",
"source": [
"### 1. Getting a GPU\n",
"\n",
"1. Easiest - Use Google Colab for a free GPU (options to upgrade as well)\n",
"2. Use your own GPU - takes a little bit of setup and requires the investment of purchasing a GPU, there's lots of options..., see this post for what option to get: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/\n",
"3. Use cloud computing - GCP, AWS, Azure, these services allow you to rent computers on the cloud and access them\n",
"\n",
"For 2, 3 PyTorch + GPU drivers (CUDA) takes a little bit of setting up, to do this, refer to PyTorch setup documentation: https://pytorch.org/get-started/locally/ "
],
"metadata": {
"id": "Upp8e_kEfvhh"
}
},
{
"cell_type": "code",
"source": [
"!nvidia-smi"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "iKzmq38ggcjS",
"outputId": "c96c8540-593d-461c-e99b-57b98e6a549f"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Sun Feb 20 00:24:35 2022 \n",
"+-----------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |\n",
"|-------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|===============================+======================+======================|\n",
"| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 32C P0 28W / 250W | 0MiB / 16280MiB | 0% Default |\n",
"| | | N/A |\n",
"+-------------------------------+----------------------+----------------------+\n",
" \n",
"+-----------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=============================================================================|\n",
"| No running processes found |\n",
"+-----------------------------------------------------------------------------+\n"
]
}
]
},
{
"cell_type": "markdown",
"source": [
"### 2. Check for GPU access with PyTorch"
],
"metadata": {
"id": "FrZo_oN_hBK4"
}
},
{
"cell_type": "code",
"source": [
"# Check for GPU access with PyTorch\n",
"import torch\n",
"torch.cuda.is_available()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "w1d1XxxvhT8h",
"outputId": "df38af09-ffbf-4fe0-ae75-a14d8bab9da3"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"True"
]
},
"metadata": {},
"execution_count": 2
}
]
},
{
"cell_type": "markdown",
"source": [
"For PyTorch since it's capable of running compute on the GPU or CPU, it's best practice to setup device agnostic code: https://pytorch.org/docs/stable/notes/cuda.html#best-practices\n",
"\n",
"E.g. run on GPU if available, else default to CPU"
],
"metadata": {
"id": "Q1apPaLTiOol"
}
},
{
"cell_type": "code",
"source": [
"# Setup device agnostic code \n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"device"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "FxxX6NWLhhxb",
"outputId": "875ab14a-5148-41d1-9691-b7401100d29d"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'cuda'"
]
},
"metadata": {},
"execution_count": 3
}
]
},
{
"cell_type": "code",
"source": [
"# Count number of devices\n",
"torch.cuda.device_count()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "nBTXPocmh1pD",
"outputId": "c973fe94-3b34-4c23-feaf-0c19a139073e"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1"
]
},
"metadata": {},
"execution_count": 4
}
]
},
{
"cell_type": "markdown",
"source": [
"## 3. Putting tensors (and models) on the GPU\n",
"\n",
"The reason we want our tensors/models on the GPU is because using a GPU results in faster computations."
],
"metadata": {
"id": "_CDYD4VHh9aL"
}
},
{
"cell_type": "code",
"source": [
"# Create a tensor (default on the CPU)\n",
"tensor = torch.tensor([1, 2, 3])\n",
"\n",
"# Tensor not on GPU\n",
"print(tensor, tensor.device)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "GlbeX_rkiyP5",
"outputId": "a9fbdbfe-c607-402f-a861-1ee6e4a1ce0e"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"tensor([1, 2, 3]) cpu\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"# Move tensor to GPU (if available)\n",
"tensor_on_gpu = tensor.to(device)\n",
"tensor_on_gpu"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1AJny5yijCeG",
"outputId": "12bbe2b0-26ca-4a04-b89e-79934a5671bd"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3], device='cuda:0')"
]
},
"metadata": {},
"execution_count": 7
}
]
},
{
"cell_type": "markdown",
"source": [
"### 4. Moving tensors back to the CPU"
],
"metadata": {
"id": "0Je1pAbGjPV1"
}
},
{
"cell_type": "code",
"source": [
"# If tensor is on GPU, can't transform it to NumPy\n",
"tensor_on_gpu.numpy()"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 185
},
"id": "6cappzQ9jl9v",
"outputId": "58c42ada-af5a-4a2c-fdd7-6850dd672222"
},
"execution_count": null,
"outputs": [
{
"output_type": "error",
"ename": "TypeError",
"evalue": "ignored",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-8-b7da913938a5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# If tensor is on GPU, can't transform it to NumPy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtensor_on_gpu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mTypeError\u001b[0m: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first."
]
}
]
},
{
"cell_type": "code",
"source": [
"# To fix the GPU tensor with NumPy issue, we can first set it to the CPU\n",
"tensor_back_on_cpu = tensor_on_gpu.cpu().numpy()\n",
"tensor_back_on_cpu"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "rL0RrzfXjs7i",
"outputId": "c0a7ca85-4909-4f39-8c9c-5de636c90ace"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"array([1, 2, 3])"
]
},
"metadata": {},
"execution_count": 11
}
]
},
{
"cell_type": "code",
"source": [
"tensor_on_gpu"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "DxRSKgKIj-FH",
"outputId": "9a083db8-06cb-4051-f9b4-1dbe04c69637"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"tensor([1, 2, 3], device='cuda:0')"
]
},
"metadata": {},
"execution_count": 13
}
]
},
{
"cell_type": "markdown",
"source": [
"## Exercises & Extra-curriculum\n",
"\n",
"See exercises for this notebook here: https://www.learnpytorch.io/00_pytorch_fundamentals/#exercises \n",
"See the template exercises notebook for this module here: https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/exercises/00_pytorch_fundamentals_exercises.ipynb "
],
"metadata": {
"id": "49zOv23ZkC-s"
}
},
{
"cell_type": "code",
"source": [
""
],
"metadata": {
"id": "1ZC_gjuYkz_B"
},
"execution_count": null,
"outputs": []
}
]
}
Reference in New Issue View Git Blame Copy Permalink