This commit is contained in:
wehub-resource-sync
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,834 @@
|
||||
# 05. PyTorch Going Modular
|
||||
|
||||
This section answers the question, "how do I turn my notebook code into Python scripts?"
|
||||
|
||||
To do so, we're going to turn the most useful code cells in [notebook 04. PyTorch Custom Datasets](https://www.learnpytorch.io/04_pytorch_custom_datasets/) into a series of Python scripts saved to a directory called [`going_modular`](https://github.com/mrdbourke/pytorch-deep-learning/tree/main/going_modular).
|
||||
|
||||
## What is going modular?
|
||||
|
||||
Going modular involves turning notebook code (from a Jupyter Notebook or Google Colab notebook) into a series of different Python scripts that offer similar functionality.
|
||||
|
||||
For example, we could turn our notebook code from a series of cells into the following Python files:
|
||||
|
||||
* `data_setup.py` - a file to prepare and download data if needed.
|
||||
* `engine.py` - a file containing various training functions.
|
||||
* `model_builder.py` or `model.py` - a file to create a PyTorch model.
|
||||
* `train.py` - a file to leverage all other files and train a target PyTorch model.
|
||||
* `utils.py` - a file dedicated to helpful utility functions.
|
||||
|
||||
> **Note:** The naming and layout of the above files will depend on your use case and code requirements. Python scripts are as general as individual notebook cells, meaning, you could create one for almost any kind of functionality.
|
||||
|
||||
## Why would you want to go modular?
|
||||
|
||||
Notebooks are fantastic for iteratively exploring and running experiments quickly.
|
||||
|
||||
However, for larger scale projects you may find Python scripts more reproducible and easier to run.
|
||||
|
||||
Though this is a debated topic, as companies like [Netflix have shown how they use notebooks for production code](https://netflixtechblog.com/notebook-innovation-591ee3221233).
|
||||
|
||||
**Production code** is code that runs to offer a service to someone or something.
|
||||
|
||||
For example, if you have an app running online that other people can access and use, the code running that app is considered **production code**.
|
||||
|
||||
And libraries like fast.ai's [`nb-dev`](https://github.com/fastai/nbdev) (short for notebook development) enable you to write whole Python libraries (including documentation) with Jupyter Notebooks.
|
||||
|
||||
### Pros and cons of notebooks vs Python scripts
|
||||
|
||||
There's arguments for both sides.
|
||||
|
||||
But this list sums up a few of the main topics.
|
||||
|
||||
| | **Pros** | **Cons** |
|
||||
| ------------- | ------------------------------------------------------ | -------------------------------------------- |
|
||||
| **Notebooks** | Easy to experiment/get started | Versioning can be hard |
|
||||
| | Easy to share (e.g. a link to a Google Colab notebook) | Hard to use only specific parts |
|
||||
| | Very visual | Text and graphics can get in the way of code |
|
||||
|
||||
| | **Pros** | **Cons** |
|
||||
| ------------------ | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
|
||||
| **Python scripts** | Can package code together (saves rewriting similar code across different notebooks) | Experimenting isn't as visual (usually have to run the whole script rather than one cell) |
|
||||
| | Can use git for versioning | |
|
||||
| | Many open source projects use scripts | |
|
||||
| | Larger projects can be run on cloud vendors (not as much support for notebooks) | |
|
||||
|
||||
### My workflow
|
||||
|
||||
I usually start machine learning projects in Jupyter/Google Colab notebooks for quick experimentation and visualization.
|
||||
|
||||
Then when I've got something working, I move the most useful pieces of code to Python scripts.
|
||||
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/05-my-workflow-for-experimenting.png" alt="one possible workflow for writing machine learning code, start with jupyter or google colab notebooks and then move to Python scripts when you've got something working." width=1000/>
|
||||
|
||||
*There are many possible workflows for writing machine learning code. Some prefer to start with scripts, others (like me) prefer to start with notebooks and go to scripts later on.*
|
||||
|
||||
### PyTorch in the wild
|
||||
|
||||
In your travels, you'll see many code repositories for PyTorch-based ML projects have instructions on how to run the PyTorch code in the form of Python scripts.
|
||||
|
||||
For example, you might be instructed to run code like the following in a terminal/command line to train a model:
|
||||
|
||||
```
|
||||
python train.py --model MODEL_NAME --batch_size BATCH_SIZE --lr LEARNING_RATE --num_epochs NUM_EPOCHS
|
||||
```
|
||||
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/05-python-train-command-line-annotated.png" alt="command line call for training a PyTorch model with different hyperparameters" width=1000/>
|
||||
|
||||
*Running a PyTorch `train.py` script on the command line with various hyperparameter settings.*
|
||||
|
||||
In this case, `train.py` is the target Python script, it'll likely contain functions to train a PyTorch model.
|
||||
|
||||
And `--model`, `--batch_size`, `--lr` and `--num_epochs` are known as argument flags.
|
||||
|
||||
You can set these to whatever values you like and if they're compatible with `train.py`, they'll work, if not, they'll error.
|
||||
|
||||
For example, let's say we wanted to train our TinyVGG model from notebook 04 for 10 epochs with a batch size of 32 and a learning rate of 0.001:
|
||||
|
||||
```
|
||||
python train.py --model tinyvgg --batch_size 32 --lr 0.001 --num_epochs 10
|
||||
```
|
||||
|
||||
You could setup any number of these argument flags in your `train.py` script to suit your needs.
|
||||
|
||||
The PyTorch blog post for training state-of-the-art computer vision models uses this style.
|
||||
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/05-training-sota-recipe.png" alt="PyTorch training script recipe for training state of the art computer vision models" width=800/>
|
||||
|
||||
*PyTorch command line training script recipe for training state-of-the-art computer vision models with 8 GPUs. Source: [PyTorch blog](https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/#the-training-recipe).*
|
||||
|
||||
## What we're going to cover
|
||||
|
||||
The main concept of this section is: **turn useful notebook code cells into reusable Python files.**
|
||||
|
||||
Doing this will save us writing the same code over and over again.
|
||||
|
||||
There are two notebooks for this section:
|
||||
|
||||
1. [**05. Going Modular: Part 1 (cell mode)**](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/going_modular/05_pytorch_going_modular_cell_mode.ipynb) - this notebook is run as a traditional Jupyter Notebook/Google Colab notebook and is a condensed version of [notebook 04](https://www.learnpytorch.io/04_pytorch_custom_datasets/).
|
||||
2. [**05. Going Modular: Part 2 (script mode)**](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/going_modular/05_pytorch_going_modular_script_mode.ipynb) - this notebook is the same as number 1 but with added functionality to turn each of the major sections into Python scripts, such as, `data_setup.py` and `train.py`.
|
||||
|
||||
The text in this document focuses on the code cells 05. Going Modular: Part 2 (script mode), the ones with `%%writefile ...` at the top.
|
||||
|
||||
### Why two parts?
|
||||
|
||||
Because sometimes the best way to learn something is to see how it *differs* from something else.
|
||||
|
||||
If you run each notebook side-by-side you'll see how they differ and that's where the key learnings are.
|
||||
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/05-notebook-cell-mode-vs-script-mode.png" alt="running cell mode notebook vs a script mode notebook" width=1000/>
|
||||
|
||||
*Running the two notebooks for section 05 side-by-side. You'll notice that the **script mode notebook has extra code cells** to turn code from the cell mode notebook into Python scripts.*
|
||||
|
||||
### What we're working towards
|
||||
|
||||
By the end of this section we want to have two things:
|
||||
|
||||
1. The ability to train the model we built in notebook 04 (Food Vision Mini) with one line of code on the command line: `python train.py`.
|
||||
2. A directory structure of reusable Python scripts, such as:
|
||||
|
||||
```
|
||||
going_modular/
|
||||
├── going_modular/
|
||||
│ ├── data_setup.py
|
||||
│ ├── engine.py
|
||||
│ ├── model_builder.py
|
||||
│ ├── train.py
|
||||
│ └── utils.py
|
||||
├── models/
|
||||
│ ├── 05_going_modular_cell_mode_tinyvgg_model.pth
|
||||
│ └── 05_going_modular_script_mode_tinyvgg_model.pth
|
||||
└── data/
|
||||
└── pizza_steak_sushi/
|
||||
├── train/
|
||||
│ ├── pizza/
|
||||
│ │ ├── image01.jpeg
|
||||
│ │ └── ...
|
||||
│ ├── steak/
|
||||
│ └── sushi/
|
||||
└── test/
|
||||
├── pizza/
|
||||
├── steak/
|
||||
└── sushi/
|
||||
```
|
||||
|
||||
### Things to note
|
||||
|
||||
* **Docstrings** - Writing reproducible and understandable code is important. And with this in mind, each of the functions/classes we'll be putting into scripts has been created with Google's [Python docstring style in mind](https://google.github.io/styleguide/pyguide.html#383-functions-and-methods).
|
||||
* **Imports at the top of scripts** - Since all of the Python scripts we're going to create could be considered a small program on their own, all of the scripts require their input modules be imported at the start of the script for example:
|
||||
|
||||
```python
|
||||
# Import modules required for train.py
|
||||
import os
|
||||
import torch
|
||||
import data_setup, engine, model_builder, utils
|
||||
|
||||
from torchvision import transforms
|
||||
```
|
||||
|
||||
## Where can you get help?
|
||||
|
||||
All of the materials for this course [are available on GitHub](https://github.com/mrdbourke/pytorch-deep-learning).
|
||||
|
||||
If you run into trouble, you can ask a question on the course [GitHub Discussions page](https://github.com/mrdbourke/pytorch-deep-learning/discussions).
|
||||
|
||||
And of course, there's the [PyTorch documentation](https://pytorch.org/docs/stable/index.html) and [PyTorch developer forums](https://discuss.pytorch.org/), a very helpful place for all things PyTorch.
|
||||
|
||||
## 0. Cell mode vs. script mode
|
||||
|
||||
A cell mode notebook such as [05. Going Modular Part 1 (cell mode)](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/going_modular/05_pytorch_going_modular_cell_mode.ipynb) is a notebook run normally, each cell in the notebook is either code or markdown.
|
||||
|
||||
A script mode notebook such as [05. Going Modular Part 2 (script mode)](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/going_modular/05_pytorch_going_modular_script_mode.ipynb) is very similar to a cell mode notebook, however, many of the code cells may be turned into Python scripts.
|
||||
|
||||
> **Note:** You don't *need* to create Python scripts via a notebook, you can create them directly through an IDE (integrated developer environment) such as [VS Code](https://code.visualstudio.com/). Having the script mode notebook as part of this section is just to demonstrate one way of going from notebooks to Python scripts.
|
||||
|
||||
## 1. Get data
|
||||
|
||||
Getting the data in each of the 05 notebooks happens the same as in [notebook 04](https://www.learnpytorch.io/04_pytorch_custom_datasets/#1-get-data).
|
||||
|
||||
A call is made to GitHub via Python's `requests` module to download a `.zip` file and unzip it.
|
||||
|
||||
```python
|
||||
import os
|
||||
import requests
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
# Setup path to data folder
|
||||
data_path = Path("data/")
|
||||
image_path = data_path / "pizza_steak_sushi"
|
||||
|
||||
# If the image folder doesn't exist, download it and prepare it...
|
||||
if image_path.is_dir():
|
||||
print(f"{image_path} directory exists.")
|
||||
else:
|
||||
print(f"Did not find {image_path} directory, creating one...")
|
||||
image_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Download pizza, steak, sushi data
|
||||
with open(data_path / "pizza_steak_sushi.zip", "wb") as f:
|
||||
request = requests.get("https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip")
|
||||
print("Downloading pizza, steak, sushi data...")
|
||||
f.write(request.content)
|
||||
|
||||
# Unzip pizza, steak, sushi data
|
||||
with zipfile.ZipFile(data_path / "pizza_steak_sushi.zip", "r") as zip_ref:
|
||||
print("Unzipping pizza, steak, sushi data...")
|
||||
zip_ref.extractall(image_path)
|
||||
|
||||
# Remove zip file
|
||||
os.remove(data_path / "pizza_steak_sushi.zip")
|
||||
```
|
||||
|
||||
This results in having a file called `data` that contains another directory called `pizza_steak_sushi` with images of pizza, steak and sushi in standard image classification format.
|
||||
|
||||
```
|
||||
data/
|
||||
└── pizza_steak_sushi/
|
||||
├── train/
|
||||
│ ├── pizza/
|
||||
│ │ ├── train_image01.jpeg
|
||||
│ │ ├── test_image02.jpeg
|
||||
│ │ └── ...
|
||||
│ ├── steak/
|
||||
│ │ └── ...
|
||||
│ └── sushi/
|
||||
│ └── ...
|
||||
└── test/
|
||||
├── pizza/
|
||||
│ ├── test_image01.jpeg
|
||||
│ └── test_image02.jpeg
|
||||
├── steak/
|
||||
└── sushi/
|
||||
```
|
||||
|
||||
## 2. Create Datasets and DataLoaders (`data_setup.py`)
|
||||
|
||||
Once we've got data, we can then turn it into PyTorch `Dataset`'s and `DataLoader`'s (one for training data and one for testing data).
|
||||
|
||||
We convert the useful `Dataset` and `DataLoader` creation code into a function called `create_dataloaders()`.
|
||||
|
||||
And we write it to file using the line `%%writefile going_modular/data_setup.py`.
|
||||
|
||||
```py title="data_setup.py"
|
||||
%%writefile going_modular/data_setup.py
|
||||
"""
|
||||
Contains functionality for creating PyTorch DataLoaders for
|
||||
image classification data.
|
||||
"""
|
||||
import os
|
||||
|
||||
from torchvision import datasets, transforms
|
||||
from torch.utils.data import DataLoader
|
||||
|
||||
NUM_WORKERS = os.cpu_count()
|
||||
|
||||
def create_dataloaders(
|
||||
train_dir: str,
|
||||
test_dir: str,
|
||||
transform: transforms.Compose,
|
||||
batch_size: int,
|
||||
num_workers: int=NUM_WORKERS
|
||||
):
|
||||
"""Creates training and testing DataLoaders.
|
||||
|
||||
Takes in a training directory and testing directory path and turns
|
||||
them into PyTorch Datasets and then into PyTorch DataLoaders.
|
||||
|
||||
Args:
|
||||
train_dir: Path to training directory.
|
||||
test_dir: Path to testing directory.
|
||||
transform: torchvision transforms to perform on training and testing data.
|
||||
batch_size: Number of samples per batch in each of the DataLoaders.
|
||||
num_workers: An integer for number of workers per DataLoader.
|
||||
|
||||
Returns:
|
||||
A tuple of (train_dataloader, test_dataloader, class_names).
|
||||
Where class_names is a list of the target classes.
|
||||
Example usage:
|
||||
train_dataloader, test_dataloader, class_names = \
|
||||
= create_dataloaders(train_dir=path/to/train_dir,
|
||||
test_dir=path/to/test_dir,
|
||||
transform=some_transform,
|
||||
batch_size=32,
|
||||
num_workers=4)
|
||||
"""
|
||||
# Use ImageFolder to create dataset(s)
|
||||
train_data = datasets.ImageFolder(train_dir, transform=transform)
|
||||
test_data = datasets.ImageFolder(test_dir, transform=transform)
|
||||
|
||||
# Get class names
|
||||
class_names = train_data.classes
|
||||
|
||||
# Turn images into data loaders
|
||||
train_dataloader = DataLoader(
|
||||
train_data,
|
||||
batch_size=batch_size,
|
||||
shuffle=True,
|
||||
num_workers=num_workers,
|
||||
pin_memory=True,
|
||||
)
|
||||
test_dataloader = DataLoader(
|
||||
test_data,
|
||||
batch_size=batch_size,
|
||||
shuffle=True,
|
||||
num_workers=num_workers,
|
||||
pin_memory=True,
|
||||
)
|
||||
|
||||
return train_dataloader, test_dataloader, class_names
|
||||
```
|
||||
|
||||
If we'd like to make `DataLoader`'s we can now use the function within `data_setup.py` like so:
|
||||
|
||||
```python
|
||||
# Import data_setup.py
|
||||
from going_modular import data_setup
|
||||
|
||||
# Create train/test dataloader and get class names as a list
|
||||
train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(...)
|
||||
```
|
||||
|
||||
## 3. Making a model (`model_builder.py`)
|
||||
|
||||
Over the past few notebooks (notebook 03 and notebook 04), we've built the TinyVGG model a few times.
|
||||
|
||||
So it makes sense to put the model into its file so we can reuse it again and again.
|
||||
|
||||
Let's put our `TinyVGG()` model class into a script with the line `%%writefile going_modular/model_builder.py`:
|
||||
|
||||
```python title="model_builder.py"
|
||||
%%writefile going_modular/model_builder.py
|
||||
"""
|
||||
Contains PyTorch model code to instantiate a TinyVGG model.
|
||||
"""
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
class TinyVGG(nn.Module):
|
||||
"""Creates the TinyVGG architecture.
|
||||
|
||||
Replicates the TinyVGG architecture from the CNN explainer website in PyTorch.
|
||||
See the original architecture here: https://poloclub.github.io/cnn-explainer/
|
||||
|
||||
Args:
|
||||
input_shape: An integer indicating number of input channels.
|
||||
hidden_units: An integer indicating number of hidden units between layers.
|
||||
output_shape: An integer indicating number of output units.
|
||||
"""
|
||||
def __init__(self, input_shape: int, hidden_units: int, output_shape: int) -> None:
|
||||
super().__init__()
|
||||
self.conv_block_1 = nn.Sequential(
|
||||
nn.Conv2d(in_channels=input_shape,
|
||||
out_channels=hidden_units,
|
||||
kernel_size=3,
|
||||
stride=1,
|
||||
padding=0),
|
||||
nn.ReLU(),
|
||||
nn.Conv2d(in_channels=hidden_units,
|
||||
out_channels=hidden_units,
|
||||
kernel_size=3,
|
||||
stride=1,
|
||||
padding=0),
|
||||
nn.ReLU(),
|
||||
nn.MaxPool2d(kernel_size=2,
|
||||
stride=2)
|
||||
)
|
||||
self.conv_block_2 = nn.Sequential(
|
||||
nn.Conv2d(hidden_units, hidden_units, kernel_size=3, padding=0),
|
||||
nn.ReLU(),
|
||||
nn.Conv2d(hidden_units, hidden_units, kernel_size=3, padding=0),
|
||||
nn.ReLU(),
|
||||
nn.MaxPool2d(2)
|
||||
)
|
||||
self.classifier = nn.Sequential(
|
||||
nn.Flatten(),
|
||||
# Where did this in_features shape come from?
|
||||
# It's because each layer of our network compresses and changes the shape of our inputs data.
|
||||
nn.Linear(in_features=hidden_units*13*13,
|
||||
out_features=output_shape)
|
||||
)
|
||||
|
||||
def forward(self, x: torch.Tensor):
|
||||
x = self.conv_block_1(x)
|
||||
x = self.conv_block_2(x)
|
||||
x = self.classifier(x)
|
||||
return x
|
||||
# return self.classifier(self.block_2(self.block_1(x))) # <- leverage the benefits of operator fusion
|
||||
```
|
||||
|
||||
Now instead of coding the TinyVGG model from scratch every time, we can import it using:
|
||||
|
||||
```python
|
||||
import torch
|
||||
# Import model_builder.py
|
||||
from going_modular import model_builder
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
# Instantiate an instance of the model from the "model_builder.py" script
|
||||
torch.manual_seed(42)
|
||||
model = model_builder.TinyVGG(input_shape=3,
|
||||
hidden_units=10,
|
||||
output_shape=len(class_names)).to(device)
|
||||
```
|
||||
|
||||
## 4. Creating `train_step()` and `test_step()` functions and `train()` to combine them
|
||||
|
||||
We wrote several training functions in [notebook 04](https://www.learnpytorch.io/04_pytorch_custom_datasets/#75-create-train-test-loop-functions):
|
||||
|
||||
1. `train_step()` - takes in a model, a `DataLoader`, a loss function and an optimizer and trains the model on the `DataLoader`.
|
||||
2. `test_step()` - takes in a model, a `DataLoader` and a loss function and evaluates the model on the `DataLoader`.
|
||||
3. `train()` - performs 1. and 2. together for a given number of epochs and returns a results dictionary.
|
||||
|
||||
Since these will be the *engine* of our model training, we can put them all into a Python script called `engine.py` with the line `%%writefile going_modular/engine.py`:
|
||||
|
||||
```python title="engine.py"
|
||||
%%writefile going_modular/engine.py
|
||||
"""
|
||||
Contains functions for training and testing a PyTorch model.
|
||||
"""
|
||||
import torch
|
||||
|
||||
from tqdm.auto import tqdm
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
def train_step(model: torch.nn.Module,
|
||||
dataloader: torch.utils.data.DataLoader,
|
||||
loss_fn: torch.nn.Module,
|
||||
optimizer: torch.optim.Optimizer,
|
||||
device: torch.device) -> Tuple[float, float]:
|
||||
"""Trains a PyTorch model for a single epoch.
|
||||
|
||||
Turns a target PyTorch model to training mode and then
|
||||
runs through all of the required training steps (forward
|
||||
pass, loss calculation, optimizer step).
|
||||
|
||||
Args:
|
||||
model: A PyTorch model to be trained.
|
||||
dataloader: A DataLoader instance for the model to be trained on.
|
||||
loss_fn: A PyTorch loss function to minimize.
|
||||
optimizer: A PyTorch optimizer to help minimize the loss function.
|
||||
device: A target device to compute on (e.g. "cuda" or "cpu").
|
||||
|
||||
Returns:
|
||||
A tuple of training loss and training accuracy metrics.
|
||||
In the form (train_loss, train_accuracy). For example:
|
||||
|
||||
(0.1112, 0.8743)
|
||||
"""
|
||||
# Put model in train mode
|
||||
model.train()
|
||||
|
||||
# Setup train loss and train accuracy values
|
||||
train_loss, train_acc = 0, 0
|
||||
|
||||
# Loop through data loader data batches
|
||||
for batch, (X, y) in enumerate(dataloader):
|
||||
# Send data to target device
|
||||
X, y = X.to(device), y.to(device)
|
||||
|
||||
# 1. Forward pass
|
||||
y_pred = model(X)
|
||||
|
||||
# 2. Calculate and accumulate loss
|
||||
loss = loss_fn(y_pred, y)
|
||||
train_loss += loss.item()
|
||||
|
||||
# 3. Optimizer zero grad
|
||||
optimizer.zero_grad()
|
||||
|
||||
# 4. Loss backward
|
||||
loss.backward()
|
||||
|
||||
# 5. Optimizer step
|
||||
optimizer.step()
|
||||
|
||||
# Calculate and accumulate accuracy metric across all batches
|
||||
y_pred_class = torch.argmax(torch.softmax(y_pred, dim=1), dim=1)
|
||||
train_acc += (y_pred_class == y).sum().item()/len(y_pred)
|
||||
|
||||
# Adjust metrics to get average loss and accuracy per batch
|
||||
train_loss = train_loss / len(dataloader)
|
||||
train_acc = train_acc / len(dataloader)
|
||||
return train_loss, train_acc
|
||||
|
||||
def test_step(model: torch.nn.Module,
|
||||
dataloader: torch.utils.data.DataLoader,
|
||||
loss_fn: torch.nn.Module,
|
||||
device: torch.device) -> Tuple[float, float]:
|
||||
"""Tests a PyTorch model for a single epoch.
|
||||
|
||||
Turns a target PyTorch model to "eval" mode and then performs
|
||||
a forward pass on a testing dataset.
|
||||
|
||||
Args:
|
||||
model: A PyTorch model to be tested.
|
||||
dataloader: A DataLoader instance for the model to be tested on.
|
||||
loss_fn: A PyTorch loss function to calculate loss on the test data.
|
||||
device: A target device to compute on (e.g. "cuda" or "cpu").
|
||||
|
||||
Returns:
|
||||
A tuple of testing loss and testing accuracy metrics.
|
||||
In the form (test_loss, test_accuracy). For example:
|
||||
|
||||
(0.0223, 0.8985)
|
||||
"""
|
||||
# Put model in eval mode
|
||||
model.eval()
|
||||
|
||||
# Setup test loss and test accuracy values
|
||||
test_loss, test_acc = 0, 0
|
||||
|
||||
# Turn on inference context manager
|
||||
with torch.inference_mode():
|
||||
# Loop through DataLoader batches
|
||||
for batch, (X, y) in enumerate(dataloader):
|
||||
# Send data to target device
|
||||
X, y = X.to(device), y.to(device)
|
||||
|
||||
# 1. Forward pass
|
||||
test_pred_logits = model(X)
|
||||
|
||||
# 2. Calculate and accumulate loss
|
||||
loss = loss_fn(test_pred_logits, y)
|
||||
test_loss += loss.item()
|
||||
|
||||
# Calculate and accumulate accuracy
|
||||
test_pred_labels = test_pred_logits.argmax(dim=1)
|
||||
test_acc += ((test_pred_labels == y).sum().item()/len(test_pred_labels))
|
||||
|
||||
# Adjust metrics to get average loss and accuracy per batch
|
||||
test_loss = test_loss / len(dataloader)
|
||||
test_acc = test_acc / len(dataloader)
|
||||
return test_loss, test_acc
|
||||
|
||||
def train(model: torch.nn.Module,
|
||||
train_dataloader: torch.utils.data.DataLoader,
|
||||
test_dataloader: torch.utils.data.DataLoader,
|
||||
optimizer: torch.optim.Optimizer,
|
||||
loss_fn: torch.nn.Module,
|
||||
epochs: int,
|
||||
device: torch.device) -> Dict[str, List]:
|
||||
"""Trains and tests a PyTorch model.
|
||||
|
||||
Passes a target PyTorch models through train_step() and test_step()
|
||||
functions for a number of epochs, training and testing the model
|
||||
in the same epoch loop.
|
||||
|
||||
Calculates, prints and stores evaluation metrics throughout.
|
||||
|
||||
Args:
|
||||
model: A PyTorch model to be trained and tested.
|
||||
train_dataloader: A DataLoader instance for the model to be trained on.
|
||||
test_dataloader: A DataLoader instance for the model to be tested on.
|
||||
optimizer: A PyTorch optimizer to help minimize the loss function.
|
||||
loss_fn: A PyTorch loss function to calculate loss on both datasets.
|
||||
epochs: An integer indicating how many epochs to train for.
|
||||
device: A target device to compute on (e.g. "cuda" or "cpu").
|
||||
|
||||
Returns:
|
||||
A dictionary of training and testing loss as well as training and
|
||||
testing accuracy metrics. Each metric has a value in a list for
|
||||
each epoch.
|
||||
In the form: {train_loss: [...],
|
||||
train_acc: [...],
|
||||
test_loss: [...],
|
||||
test_acc: [...]}
|
||||
For example if training for epochs=2:
|
||||
{train_loss: [2.0616, 1.0537],
|
||||
train_acc: [0.3945, 0.3945],
|
||||
test_loss: [1.2641, 1.5706],
|
||||
test_acc: [0.3400, 0.2973]}
|
||||
"""
|
||||
# Create empty results dictionary
|
||||
results = {"train_loss": [],
|
||||
"train_acc": [],
|
||||
"test_loss": [],
|
||||
"test_acc": []
|
||||
}
|
||||
|
||||
# Loop through training and testing steps for a number of epochs
|
||||
for epoch in tqdm(range(epochs)):
|
||||
train_loss, train_acc = train_step(model=model,
|
||||
dataloader=train_dataloader,
|
||||
loss_fn=loss_fn,
|
||||
optimizer=optimizer,
|
||||
device=device)
|
||||
test_loss, test_acc = test_step(model=model,
|
||||
dataloader=test_dataloader,
|
||||
loss_fn=loss_fn,
|
||||
device=device)
|
||||
|
||||
# Print out what's happening
|
||||
print(
|
||||
f"Epoch: {epoch+1} | "
|
||||
f"train_loss: {train_loss:.4f} | "
|
||||
f"train_acc: {train_acc:.4f} | "
|
||||
f"test_loss: {test_loss:.4f} | "
|
||||
f"test_acc: {test_acc:.4f}"
|
||||
)
|
||||
|
||||
# Update results dictionary
|
||||
results["train_loss"].append(train_loss)
|
||||
results["train_acc"].append(train_acc)
|
||||
results["test_loss"].append(test_loss)
|
||||
results["test_acc"].append(test_acc)
|
||||
|
||||
# Return the filled results at the end of the epochs
|
||||
return results
|
||||
```
|
||||
|
||||
Now we've got the `engine.py` script, we can import functions from it via:
|
||||
|
||||
```python
|
||||
# Import engine.py
|
||||
from going_modular import engine
|
||||
|
||||
# Use train() by calling it from engine.py
|
||||
engine.train(...)
|
||||
```
|
||||
|
||||
## 5. Creating a function to save the model (`utils.py`)
|
||||
|
||||
Often you'll want to save a model whilst it's training or after training.
|
||||
|
||||
Since we've written the code to save a model a few times now in previous notebooks, it makes sense to turn it into a function and save it to file.
|
||||
|
||||
It's common practice to store helper functions in a file called `utils.py` (short for utilities).
|
||||
|
||||
Let's save our `save_model()` function to a file called `utils.py` with the line `%%writefile going_modular/utils.py`:
|
||||
|
||||
```python title="utils.py"
|
||||
%%writefile going_modular/utils.py
|
||||
"""
|
||||
Contains various utility functions for PyTorch model training and saving.
|
||||
"""
|
||||
import torch
|
||||
from pathlib import Path
|
||||
|
||||
def save_model(model: torch.nn.Module,
|
||||
target_dir: str,
|
||||
model_name: str):
|
||||
"""Saves a PyTorch model to a target directory.
|
||||
|
||||
Args:
|
||||
model: A target PyTorch model to save.
|
||||
target_dir: A directory for saving the model to.
|
||||
model_name: A filename for the saved model. Should include
|
||||
either ".pth" or ".pt" as the file extension.
|
||||
|
||||
Example usage:
|
||||
save_model(model=model_0,
|
||||
target_dir="models",
|
||||
model_name="05_going_modular_tingvgg_model.pth")
|
||||
"""
|
||||
# Create target directory
|
||||
target_dir_path = Path(target_dir)
|
||||
target_dir_path.mkdir(parents=True,
|
||||
exist_ok=True)
|
||||
|
||||
# Create model save path
|
||||
assert model_name.endswith(".pth") or model_name.endswith(".pt"), "model_name should end with '.pt' or '.pth'"
|
||||
model_save_path = target_dir_path / model_name
|
||||
|
||||
# Save the model state_dict()
|
||||
print(f"[INFO] Saving model to: {model_save_path}")
|
||||
torch.save(obj=model.state_dict(),
|
||||
f=model_save_path)
|
||||
```
|
||||
|
||||
Now if we wanted to use our `save_model()` function, instead of writing it all over again, we can import it and use it via:
|
||||
|
||||
```python
|
||||
# Import utils.py
|
||||
from going_modular import utils
|
||||
|
||||
# Save a model to file
|
||||
save_model(model=...
|
||||
target_dir=...,
|
||||
model_name=...)
|
||||
```
|
||||
|
||||
## 6. Train, evaluate and save the model (`train.py`)
|
||||
|
||||
As previously discussed, you'll often come across PyTorch repositories that combine all of their functionality together in a `train.py` file.
|
||||
|
||||
This file is essentially saying "train the model using whatever data is available".
|
||||
|
||||
In our `train.py` file, we'll combine all of the functionality of the other Python scripts we've created and use it to train a model.
|
||||
|
||||
This way we can train a PyTorch model using a single line of code on the command line:
|
||||
|
||||
```
|
||||
python train.py
|
||||
```
|
||||
|
||||
To create `train.py` we'll go through the following steps:
|
||||
|
||||
1. Import the various dependencies, namely `torch`, `os`, `torchvision.transforms` and all of the scripts from the `going_modular` directory, `data_setup`, `engine`, `model_builder`, `utils`.
|
||||
* **Note:** Since `train.py` will be *inside* the `going_modular` directory, we can import the other modules via `import ...` rather than `from going_modular import ...`.
|
||||
2. Setup various hyperparameters such as batch size, number of epochs, learning rate and number of hidden units (these could be set in the future via [Python's `argparse`](https://docs.python.org/3/library/argparse.html)).
|
||||
3. Setup the training and test directories.
|
||||
4. Setup device-agnostic code.
|
||||
5. Create the necessary data transforms.
|
||||
6. Create the DataLoaders using `data_setup.py`.
|
||||
7. Create the model using `model_builder.py`.
|
||||
8. Setup the loss function and optimizer.
|
||||
9. Train the model using `engine.py`.
|
||||
10. Save the model using `utils.py`.
|
||||
|
||||
And we can create the file from a notebook cell using the line `%%writefile going_modular/train.py`:
|
||||
|
||||
```python title="train.py"
|
||||
%%writefile going_modular/train.py
|
||||
"""
|
||||
Trains a PyTorch image classification model using device-agnostic code.
|
||||
"""
|
||||
|
||||
import os
|
||||
import torch
|
||||
import data_setup, engine, model_builder, utils
|
||||
|
||||
from torchvision import transforms
|
||||
|
||||
# Setup hyperparameters
|
||||
NUM_EPOCHS = 5
|
||||
BATCH_SIZE = 32
|
||||
HIDDEN_UNITS = 10
|
||||
LEARNING_RATE = 0.001
|
||||
|
||||
# Setup directories
|
||||
train_dir = "data/pizza_steak_sushi/train"
|
||||
test_dir = "data/pizza_steak_sushi/test"
|
||||
|
||||
# Setup target device
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
# Create transforms
|
||||
data_transform = transforms.Compose([
|
||||
transforms.Resize((64, 64)),
|
||||
transforms.ToTensor()
|
||||
])
|
||||
|
||||
# Create DataLoaders with help from data_setup.py
|
||||
train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(
|
||||
train_dir=train_dir,
|
||||
test_dir=test_dir,
|
||||
transform=data_transform,
|
||||
batch_size=BATCH_SIZE
|
||||
)
|
||||
|
||||
# Create model with help from model_builder.py
|
||||
model = model_builder.TinyVGG(
|
||||
input_shape=3,
|
||||
hidden_units=HIDDEN_UNITS,
|
||||
output_shape=len(class_names)
|
||||
).to(device)
|
||||
|
||||
# Set loss and optimizer
|
||||
loss_fn = torch.nn.CrossEntropyLoss()
|
||||
optimizer = torch.optim.Adam(model.parameters(),
|
||||
lr=LEARNING_RATE)
|
||||
|
||||
# Start training with help from engine.py
|
||||
engine.train(model=model,
|
||||
train_dataloader=train_dataloader,
|
||||
test_dataloader=test_dataloader,
|
||||
loss_fn=loss_fn,
|
||||
optimizer=optimizer,
|
||||
epochs=NUM_EPOCHS,
|
||||
device=device)
|
||||
|
||||
# Save the model with help from utils.py
|
||||
utils.save_model(model=model,
|
||||
target_dir="models",
|
||||
model_name="05_going_modular_script_mode_tinyvgg_model.pth")
|
||||
```
|
||||
|
||||
Woohoo!
|
||||
|
||||
Now we can train a PyTorch model by running the following line on the command line:
|
||||
|
||||
```
|
||||
python train.py
|
||||
```
|
||||
|
||||
Doing this will leverage all of the other code scripts we've created.
|
||||
|
||||
And if we wanted to, we could adjust our `train.py` file to use argument flag inputs with Python's `argparse` module, this would allow us to provide different hyperparameter settings like previously discussed:
|
||||
|
||||
```
|
||||
python train.py --model MODEL_NAME --batch_size BATCH_SIZE --lr LEARNING_RATE --num_epochs NUM_EPOCHS
|
||||
```
|
||||
|
||||
## Exercises
|
||||
|
||||
**Resources:**
|
||||
|
||||
* [Exercise template notebook for 05](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/exercises/05_pytorch_going_modular_exercise_template.ipynb)
|
||||
* [Example solutions notebook for 05](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/solutions/05_pytorch_going_modular_exercise_solutions.ipynb)
|
||||
* Live coding run through of [solutions notebook for 05 on YouTube](https://youtu.be/ijgFhMK3pp4)
|
||||
|
||||
**Exercises:**
|
||||
|
||||
1. Turn the code to get the data (from section 1. Get Data above) into a Python script, such as `get_data.py`.
|
||||
* When you run the script using `python get_data.py` it should check if the data already exists and skip downloading if it does.
|
||||
* If the data download is successful, you should be able to access the `pizza_steak_sushi` images from the `data` directory.
|
||||
2. Use [Python's `argparse` module](https://docs.python.org/3/library/argparse.html) to be able to send the `train.py` custom hyperparameter values for training procedures.
|
||||
* Add an argument for using a different:
|
||||
* Training/testing directory
|
||||
* Learning rate
|
||||
* Batch size
|
||||
* Number of epochs to train for
|
||||
* Number of hidden units in the TinyVGG model
|
||||
* Keep the default values for each of the above arguments as what they already are (as in notebook 05).
|
||||
* For example, you should be able to run something similar to the following line to train a TinyVGG model with a learning rate of 0.003 and a batch size of 64 for 20 epochs: `python train.py --learning_rate 0.003 --batch_size 64 --num_epochs 20`.
|
||||
* **Note:** Since `train.py` leverages the other scripts we created in section 05, such as, `model_builder.py`, `utils.py` and `engine.py`, you'll have to make sure they're available to use too. You can find these in the [`going_modular` folder on the course GitHub](https://github.com/mrdbourke/pytorch-deep-learning/tree/main/going_modular/going_modular).
|
||||
3. Create a script to predict (such as `predict.py`) on a target image given a file path with a saved model.
|
||||
* For example, you should be able to run the command `python predict.py some_image.jpeg` and have a trained PyTorch model predict on the image and return its prediction.
|
||||
* To see example prediction code, check out the [predicting on a custom image section in notebook 04](https://www.learnpytorch.io/04_pytorch_custom_datasets/#113-putting-custom-image-prediction-together-building-a-function).
|
||||
* You may also have to write code to load in a trained model.
|
||||
|
||||
## Extra-curriculum
|
||||
|
||||
* To learn more about structuring a Python project, check out Real Python's guide on [Python Application Layouts](https://realpython.com/python-application-layouts/).
|
||||
* For ideas on styling your PyTorch code, check out the [PyTorch style guide by Igor Susmelj](https://github.com/IgorSusmelj/pytorch-styleguide#recommended-code-structure-for-training-your-model) (much of styling in this chapter is based off this guide + various similar PyTorch repositories).
|
||||
* For an example `train.py` script and various other PyTorch scripts written by the PyTorch team to train state-of-the-art image classification models, check out their [`classification` repository on GitHub](https://github.com/pytorch/vision/tree/main/references/classification).
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,691 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 08: PyTorch Profiling\n",
|
||||
"\n",
|
||||
"This notebook is an experiment to try out the PyTorch profiler.\n",
|
||||
"\n",
|
||||
"See here for more:\n",
|
||||
"* https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/\n",
|
||||
"* https://pytorch.org/docs/stable/profiler.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchvision\n",
|
||||
"from torch import nn\n",
|
||||
"from torchvision import transforms, datasets\n",
|
||||
"from torchinfo import summary\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"from going_modular import data_setup, engine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Setup device"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'cuda'"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
|
||||
"device"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get and load data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"data/10_whole_foods dir exists, skipping download\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import requests\n",
|
||||
"from zipfile import ZipFile\n",
|
||||
"\n",
|
||||
"def get_food_image_data():\n",
|
||||
" if not os.path.exists(\"data/10_whole_foods\"):\n",
|
||||
" os.makedirs(\"data/\", exist_ok=True)\n",
|
||||
" # Download data\n",
|
||||
" data_url = \"https://storage.googleapis.com/food-vision-image-playground/10_whole_foods.zip\"\n",
|
||||
" print(f\"Downloading data from {data_url}...\")\n",
|
||||
" requests.get(data_url)\n",
|
||||
" # Unzip data\n",
|
||||
" targ_dir = \"data/10_whole_foods\"\n",
|
||||
" print(f\"Extracting data to {targ_dir}...\")\n",
|
||||
" with ZipFile(\"10_whole_foods.zip\") as zip_ref:\n",
|
||||
" zip_ref.extractall(targ_dir)\n",
|
||||
" else:\n",
|
||||
" print(\"data/10_whole_foods dir exists, skipping download\")\n",
|
||||
"\n",
|
||||
"get_food_image_data()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(<torch.utils.data.dataloader.DataLoader at 0x7f052ab26e20>,\n",
|
||||
" <torch.utils.data.dataloader.DataLoader at 0x7f05299eed00>,\n",
|
||||
" ['apple',\n",
|
||||
" 'banana',\n",
|
||||
" 'beef',\n",
|
||||
" 'blueberries',\n",
|
||||
" 'carrots',\n",
|
||||
" 'chicken_wings',\n",
|
||||
" 'egg',\n",
|
||||
" 'honey',\n",
|
||||
" 'mushrooms',\n",
|
||||
" 'strawberries'])"
|
||||
]
|
||||
},
|
||||
"execution_count": 38,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Setup dirs\n",
|
||||
"train_dir = \"data/10_whole_foods/train\"\n",
|
||||
"test_dir = \"data/10_whole_foods/test\"\n",
|
||||
"\n",
|
||||
"# Setup ImageNet normalization levels (turns all images into similar distribution as ImageNet)\n",
|
||||
"normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
|
||||
" std=[0.229, 0.224, 0.225])\n",
|
||||
"\n",
|
||||
"# Create starter transform\n",
|
||||
"simple_transform = transforms.Compose([\n",
|
||||
" transforms.Resize((224, 224)),\n",
|
||||
" transforms.ToTensor(),\n",
|
||||
" normalize\n",
|
||||
"]) \n",
|
||||
"\n",
|
||||
"# Create data loaders\n",
|
||||
"train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(\n",
|
||||
" train_dir=train_dir,\n",
|
||||
" test_dir=test_dir,\n",
|
||||
" transform=simple_transform,\n",
|
||||
" batch_size=32,\n",
|
||||
" num_workers=8\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"train_dataloader, test_dataloader, class_names"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Load model "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = torchvision.models.efficientnet_b0(pretrained=True).to(device)\n",
|
||||
"# model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Update the classifier\n",
|
||||
"model.classifier = torch.nn.Sequential(\n",
|
||||
" nn.Dropout(p=0.2),\n",
|
||||
" nn.Linear(1280, len(class_names)).to(device))\n",
|
||||
"\n",
|
||||
"# Freeze all base layers \n",
|
||||
"for param in model.features.parameters():\n",
|
||||
" param.requires_grad = False"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Train model and track results"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Define loss and optimizer\n",
|
||||
"loss_fn = nn.CrossEntropyLoss()\n",
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Adjust training function to track results with `SummaryWriter`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 69,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'EfficietNetB0'"
|
||||
]
|
||||
},
|
||||
"execution_count": 69,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.name = \"EfficietNetB0\"\n",
|
||||
"model.name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 70,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.tensorboard import SummaryWriter\n",
|
||||
"from going_modular.engine import train_step, test_step\n",
|
||||
"from tqdm import tqdm\n",
|
||||
"writer = SummaryWriter()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Update the `train_step()` function to include the PyTorch profiler."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 71,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_step(model, dataloader, loss_fn, optimizer):\n",
|
||||
" model.train()\n",
|
||||
" train_loss, train_acc = 0, 0\n",
|
||||
" ## NEW: Add PyTorch profiler\n",
|
||||
"\n",
|
||||
" dir_to_save_logs = os.path.join(\"logs\", datetime.now().strftime(\"%Y-%m-%d-%H-%M\"))\n",
|
||||
" with torch.profiler.profile(\n",
|
||||
" on_trace_ready=torch.profiler.tensorboard_trace_handler(dir_name=dir_to_save_logs),\n",
|
||||
" # with_stack=True # this adds a lot of overhead to training (tracing all the stack)\n",
|
||||
" ):\n",
|
||||
" for batch, (X, y) in enumerate(dataloader):\n",
|
||||
" # Send data to GPU\n",
|
||||
" X, y = X.to(device, non_blocking=True), y.to(device, non_blocking=True)\n",
|
||||
" \n",
|
||||
" # Turn on mixed precision if available\n",
|
||||
" with torch.autocast(device_type=device, enabled=True):\n",
|
||||
" # 1. Forward pass\n",
|
||||
" y_pred = model(X)\n",
|
||||
"\n",
|
||||
" # 2. Calculate loss\n",
|
||||
" loss = loss_fn(y_pred, y)\n",
|
||||
"\n",
|
||||
" # 3. Optimizer zero grad\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
"\n",
|
||||
" # 4. Loss backward\n",
|
||||
" loss.backward()\n",
|
||||
"\n",
|
||||
" # 5. Optimizer step\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
" # 6. Calculate metrics\n",
|
||||
" train_loss += loss.item()\n",
|
||||
" y_pred_class = torch.softmax(y_pred, dim=1).argmax(dim=1)\n",
|
||||
" # print(f\"y: \\n{y}\\ny_pred_class:{y_pred_class}\")\n",
|
||||
" # print(f\"y argmax: {y_pred.argmax(dim=1)}\")\n",
|
||||
" # print(f\"Equal: {(y_pred_class == y)}\")\n",
|
||||
" train_acc += (y_pred_class == y).sum().item() / len(y_pred)\n",
|
||||
" # print(f\"batch: {batch} train_acc: {train_acc}\")\n",
|
||||
"\n",
|
||||
" # Adjust returned metrics\n",
|
||||
" return train_loss / len(dataloader), train_acc / len(dataloader)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"TK - Now to use the writer, we've got to adjust the `train()` function..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 72,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train(\n",
|
||||
" model,\n",
|
||||
" train_dataloader,\n",
|
||||
" test_dataloader,\n",
|
||||
" optimizer,\n",
|
||||
" loss_fn=nn.CrossEntropyLoss(),\n",
|
||||
" epochs=5,\n",
|
||||
"):\n",
|
||||
"\n",
|
||||
" results = {\"train_loss\": [], \"train_acc\": [], \"test_loss\": [], \"test_acc\": []}\n",
|
||||
"\n",
|
||||
" for epoch in tqdm(range(epochs)):\n",
|
||||
" train_loss, train_acc = train_step(\n",
|
||||
" model=model,\n",
|
||||
" dataloader=train_dataloader,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" )\n",
|
||||
" test_loss, test_acc = test_step(\n",
|
||||
" model=model, dataloader=test_dataloader, loss_fn=loss_fn\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Print out what's happening\n",
|
||||
" print(\n",
|
||||
" f\"Epoch: {epoch+1} | \"\n",
|
||||
" f\"train_loss: {train_loss:.4f} | \"\n",
|
||||
" f\"train_acc: {train_acc:.4f} | \"\n",
|
||||
" f\"test_loss: {test_loss:.4f} | \"\n",
|
||||
" f\"test_acc: {test_acc:.4f}\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Update results\n",
|
||||
" results[\"train_loss\"].append(train_loss)\n",
|
||||
" results[\"train_acc\"].append(train_acc)\n",
|
||||
" results[\"test_loss\"].append(test_loss)\n",
|
||||
" results[\"test_acc\"].append(test_acc)\n",
|
||||
"\n",
|
||||
" # Add results to SummaryWriter\n",
|
||||
" writer.add_scalars(main_tag=\"Loss\", \n",
|
||||
" tag_scalar_dict={\"train_loss\": train_loss,\n",
|
||||
" \"test_loss\": test_loss},\n",
|
||||
" global_step=epoch)\n",
|
||||
" writer.add_scalars(main_tag=\"Accuracy\", \n",
|
||||
" tag_scalar_dict={\"train_acc\": train_acc,\n",
|
||||
" \"test_acc\": test_acc}, \n",
|
||||
" global_step=epoch)\n",
|
||||
" \n",
|
||||
" # Close the writer\n",
|
||||
" writer.close()\n",
|
||||
"\n",
|
||||
" return results"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 73,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 20%|██ | 1/5 [00:05<00:21, 5.27s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 1 | train_loss: 1.9644 | train_acc: 0.4386 | test_loss: 1.5205 | test_acc: 0.7865\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 40%|████ | 2/5 [00:09<00:14, 4.94s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 2 | train_loss: 1.2589 | train_acc: 0.7878 | test_loss: 1.1589 | test_acc: 0.7604\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 60%|██████ | 3/5 [00:14<00:09, 4.72s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 3 | train_loss: 0.8642 | train_acc: 0.8776 | test_loss: 0.9347 | test_acc: 0.7917\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 80%|████████ | 4/5 [00:18<00:04, 4.56s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 4 | train_loss: 0.6827 | train_acc: 0.8856 | test_loss: 0.6637 | test_acc: 0.8750\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|██████████| 5/5 [00:23<00:00, 4.65s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 5 | train_loss: 0.5688 | train_acc: 0.9069 | test_loss: 0.6175 | test_acc: 0.8854\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Train model\n",
|
||||
"# Note: Not using engine.train() since the original script isn't updated\n",
|
||||
"results = train(model=model,\n",
|
||||
" train_dataloader=train_dataloader,\n",
|
||||
" test_dataloader=test_dataloader,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Looks like mixed precision doesn't offer much benefit for smaller feature extraction models..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# # Without mixed precision\n",
|
||||
"# 20%|██ | 1/5 [00:03<00:14, 3.71s/it]Epoch: 1 | train_loss: 0.5229 | train_acc: 0.9054 | test_loss: 0.5776 | test_acc: 0.8542\n",
|
||||
"# 40%|████ | 2/5 [00:07<00:11, 3.74s/it]Epoch: 2 | train_loss: 0.4699 | train_acc: 0.9001 | test_loss: 0.5160 | test_acc: 0.8802\n",
|
||||
"# 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]Epoch: 3 | train_loss: 0.3913 | train_acc: 0.9196 | test_loss: 0.4888 | test_acc: 0.8906\n",
|
||||
"# 80%|████████ | 4/5 [00:14<00:03, 3.61s/it]Epoch: 4 | train_loss: 0.3724 | train_acc: 0.9371 | test_loss: 0.4931 | test_acc: 0.8698\n",
|
||||
"# 100%|██████████| 5/5 [00:18<00:00, 3.61s/it]Epoch: 5 | train_loss: 0.3315 | train_acc: 0.9381 | test_loss: 0.4405 | test_acc: 0.8750\n",
|
||||
"\n",
|
||||
"# # With mixed precision\n",
|
||||
"# 20%|██ | 1/5 [00:04<00:17, 4.40s/it]Epoch: 1 | train_loss: 0.3027 | train_acc: 0.9554 | test_loss: 0.4386 | test_acc: 0.8802\n",
|
||||
"# 40%|████ | 2/5 [00:08<00:13, 4.49s/it]Epoch: 2 | train_loss: 0.2826 | train_acc: 0.9539 | test_loss: 0.4080 | test_acc: 0.8802\n",
|
||||
"# 60%|██████ | 3/5 [00:13<00:08, 4.48s/it]Epoch: 3 | train_loss: 0.2450 | train_acc: 0.9609 | test_loss: 0.4130 | test_acc: 0.8750\n",
|
||||
"# 80%|████████ | 4/5 [00:18<00:04, 4.53s/it]Epoch: 4 | train_loss: 0.2450 | train_acc: 0.9594 | test_loss: 0.4158 | test_acc: 0.8802\n",
|
||||
"# 100%|██████████| 5/5 [00:22<00:00, 4.49s/it]Epoch: 5 | train_loss: 0.2307 | train_acc: 0.9639 | test_loss: 0.4124 | test_acc: 0.8906"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Try mixed precision with larger model\n",
|
||||
"\n",
|
||||
"Now we'll try turn on mixed precision with a larger model (e.g. EffifientNetB0 with all layers tuneable)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 74,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Unfreeze all base layers \n",
|
||||
"for param in model.features.parameters():\n",
|
||||
" param.requires_grad = True\n",
|
||||
"\n",
|
||||
"# for param in model.features.parameters():\n",
|
||||
"# print(param.requires_grad)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 75,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 20%|██ | 1/5 [00:13<00:53, 13.27s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 1 | train_loss: 0.4934 | train_acc: 0.8586 | test_loss: 0.6467 | test_acc: 0.7969\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 40%|████ | 2/5 [00:27<00:42, 14.09s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 2 | train_loss: 0.1750 | train_acc: 0.9628 | test_loss: 1.1806 | test_acc: 0.8385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 60%|██████ | 3/5 [00:42<00:28, 14.28s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 3 | train_loss: 0.1362 | train_acc: 0.9619 | test_loss: 0.5831 | test_acc: 0.8802\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 80%|████████ | 4/5 [00:57<00:14, 14.47s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 4 | train_loss: 0.1743 | train_acc: 0.9462 | test_loss: 0.5702 | test_acc: 0.8854\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|██████████| 5/5 [01:11<00:00, 14.38s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 5 | train_loss: 0.2437 | train_acc: 0.9352 | test_loss: 0.7096 | test_acc: 0.8125\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Train model\n",
|
||||
"# Note: Not using engine.train() since the original script isn't updated\n",
|
||||
"results = train(model=model,\n",
|
||||
" train_dataloader=train_dataloader,\n",
|
||||
" test_dataloader=test_dataloader,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# # Without mixed precision...\n",
|
||||
"# 20%|██ | 1/5 [00:11<00:46, 11.61s/it]Epoch: 1 | train_loss: 0.4507 | train_acc: 0.8648 | test_loss: 1.0603 | test_acc: 0.7604\n",
|
||||
"# 40%|████ | 2/5 [00:24<00:36, 12.21s/it]Epoch: 2 | train_loss: 0.1659 | train_acc: 0.9464 | test_loss: 0.6398 | test_acc: 0.8490\n",
|
||||
"# 60%|██████ | 3/5 [00:36<00:24, 12.38s/it]Epoch: 3 | train_loss: 0.1261 | train_acc: 0.9698 | test_loss: 0.7149 | test_acc: 0.8542\n",
|
||||
"# 80%|████████ | 4/5 [00:49<00:12, 12.53s/it]Epoch: 4 | train_loss: 0.1250 | train_acc: 0.9609 | test_loss: 0.7441 | test_acc: 0.7917\n",
|
||||
"# 100%|██████████| 5/5 [01:02<00:00, 12.42s/it]Epoch: 5 | train_loss: 0.1282 | train_acc: 0.9564 | test_loss: 0.8701 | test_acc: 0.8385\n",
|
||||
"\n",
|
||||
"# # With mixed precision...\n",
|
||||
"# 20%|██ | 1/5 [00:13<00:53, 13.27s/it]Epoch: 1 | train_loss: 0.4934 | train_acc: 0.8586 | test_loss: 0.6467 | test_acc: 0.7969\n",
|
||||
"# 40%|████ | 2/5 [00:27<00:42, 14.09s/it]Epoch: 2 | train_loss: 0.1750 | train_acc: 0.9628 | test_loss: 1.1806 | test_acc: 0.8385\n",
|
||||
"# 60%|██████ | 3/5 [00:42<00:28, 14.28s/it]Epoch: 3 | train_loss: 0.1362 | train_acc: 0.9619 | test_loss: 0.5831 | test_acc: 0.8802\n",
|
||||
"# 80%|████████ | 4/5 [00:57<00:14, 14.47s/it]Epoch: 4 | train_loss: 0.1743 | train_acc: 0.9462 | test_loss: 0.5702 | test_acc: 0.8854\n",
|
||||
"# 100%|██████████| 5/5 [01:11<00:00, 14.38s/it]Epoch: 5 | train_loss: 0.2437 | train_acc: 0.9352 | test_loss: 0.7096 | test_acc: 0.8125"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Checking the PyTorch profiler, it seems that mixed precision utilises some Tensor Cores, however, these aren't large numbers.\n",
|
||||
"\n",
|
||||
"E.g. it uses 9-12% Tensor Cores. Perhaps the slow down when using mixed precision is because the tensors have to get altered and converted when there isn't very many of them. For example only 9-12% of tensors get converted so the speed up gains aren't realised on these tensors because they get cancelled out by the conversion time."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Extensions\n",
|
||||
"* Does changing the data input size to EfficientNetB4 change its results? E.g. input image size of (380, 380) instead of (224, 224)?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
" "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "3fbe1355223f7b2ffc113ba3ade6a2b520cadace5d5ec3e828c83ce02eb221bf"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3.9.7 64-bit (conda)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.7"
|
||||
},
|
||||
"orig_nbformat": 4
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1,690 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# 08: PyTorch Profiling\n",
|
||||
"\n",
|
||||
"This notebook is an experiment to try out the PyTorch profiler.\n",
|
||||
"\n",
|
||||
"See here for more:\n",
|
||||
"* https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/\n",
|
||||
"* https://pytorch.org/docs/stable/profiler.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"import torchvision\n",
|
||||
"from torch import nn\n",
|
||||
"from torchvision import transforms, datasets\n",
|
||||
"from torchinfo import summary\n",
|
||||
"\n",
|
||||
"import numpy as np\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"from going_modular import data_setup, engine"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Setup device"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'cuda'"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
|
||||
"device"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get and load data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"data/10_whole_foods dir exists, skipping download\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"import requests\n",
|
||||
"from zipfile import ZipFile\n",
|
||||
"\n",
|
||||
"def get_food_image_data():\n",
|
||||
" if not os.path.exists(\"data/10_whole_foods\"):\n",
|
||||
" os.makedirs(\"data/\", exist_ok=True)\n",
|
||||
" # Download data\n",
|
||||
" data_url = \"https://storage.googleapis.com/food-vision-image-playground/10_whole_foods.zip\"\n",
|
||||
" print(f\"Downloading data from {data_url}...\")\n",
|
||||
" requests.get(data_url)\n",
|
||||
" # Unzip data\n",
|
||||
" targ_dir = \"data/10_whole_foods\"\n",
|
||||
" print(f\"Extracting data to {targ_dir}...\")\n",
|
||||
" with ZipFile(\"10_whole_foods.zip\") as zip_ref:\n",
|
||||
" zip_ref.extractall(targ_dir)\n",
|
||||
" else:\n",
|
||||
" print(\"data/10_whole_foods dir exists, skipping download\")\n",
|
||||
"\n",
|
||||
"get_food_image_data()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"(<torch.utils.data.dataloader.DataLoader at 0x7f052ab26e20>,\n",
|
||||
" <torch.utils.data.dataloader.DataLoader at 0x7f05299eed00>,\n",
|
||||
" ['apple',\n",
|
||||
" 'banana',\n",
|
||||
" 'beef',\n",
|
||||
" 'blueberries',\n",
|
||||
" 'carrots',\n",
|
||||
" 'chicken_wings',\n",
|
||||
" 'egg',\n",
|
||||
" 'honey',\n",
|
||||
" 'mushrooms',\n",
|
||||
" 'strawberries'])"
|
||||
]
|
||||
},
|
||||
"execution_count": 38,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Setup dirs\n",
|
||||
"train_dir = \"data/10_whole_foods/train\"\n",
|
||||
"test_dir = \"data/10_whole_foods/test\"\n",
|
||||
"\n",
|
||||
"# Setup ImageNet normalization levels (turns all images into similar distribution as ImageNet)\n",
|
||||
"normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
|
||||
" std=[0.229, 0.224, 0.225])\n",
|
||||
"\n",
|
||||
"# Create starter transform\n",
|
||||
"simple_transform = transforms.Compose([\n",
|
||||
" transforms.Resize((224, 224)),\n",
|
||||
" transforms.ToTensor(),\n",
|
||||
" normalize\n",
|
||||
"]) \n",
|
||||
"\n",
|
||||
"# Create data loaders\n",
|
||||
"train_dataloader, test_dataloader, class_names = data_setup.create_dataloaders(\n",
|
||||
" train_dir=train_dir,\n",
|
||||
" test_dir=test_dir,\n",
|
||||
" transform=simple_transform,\n",
|
||||
" batch_size=32,\n",
|
||||
" num_workers=8\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"train_dataloader, test_dataloader, class_names"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Load model "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model = torchvision.models.efficientnet_b0(pretrained=True).to(device)\n",
|
||||
"# model"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Update the classifier\n",
|
||||
"model.classifier = torch.nn.Sequential(\n",
|
||||
" nn.Dropout(p=0.2),\n",
|
||||
" nn.Linear(1280, len(class_names)).to(device))\n",
|
||||
"\n",
|
||||
"# Freeze all base layers \n",
|
||||
"for param in model.features.parameters():\n",
|
||||
" param.requires_grad = False"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Train model and track results"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Define loss and optimizer\n",
|
||||
"loss_fn = nn.CrossEntropyLoss()\n",
|
||||
"optimizer = torch.optim.Adam(model.parameters(), lr=0.001)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Adjust training function to track results with `SummaryWriter`"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 69,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'EfficietNetB0'"
|
||||
]
|
||||
},
|
||||
"execution_count": 69,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.name = \"EfficietNetB0\"\n",
|
||||
"model.name"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 70,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from torch.utils.tensorboard import SummaryWriter\n",
|
||||
"from going_modular.engine import train_step, test_step\n",
|
||||
"from tqdm import tqdm\n",
|
||||
"writer = SummaryWriter()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Update the `train_step()` function to include the PyTorch profiler."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 71,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train_step(model, dataloader, loss_fn, optimizer):\n",
|
||||
" model.train()\n",
|
||||
" train_loss, train_acc = 0, 0\n",
|
||||
" ## NEW: Add PyTorch profiler\n",
|
||||
"\n",
|
||||
" dir_to_save_logs = os.path.join(\"logs\", datetime.now().strftime(\"%Y-%m-%d-%H-%M\"))\n",
|
||||
" with torch.profiler.profile(\n",
|
||||
" on_trace_ready=torch.profiler.tensorboard_trace_handler(dir_name=dir_to_save_logs),\n",
|
||||
" # with_stack=True # this adds a lot of overhead to training (tracing all the stack)\n",
|
||||
" ):\n",
|
||||
" for batch, (X, y) in enumerate(dataloader):\n",
|
||||
" # Send data to GPU\n",
|
||||
" X, y = X.to(device, non_blocking=True), y.to(device, non_blocking=True)\n",
|
||||
" \n",
|
||||
" # Turn on mixed precision if available\n",
|
||||
" with torch.autocast(device_type=device, enabled=True):\n",
|
||||
" # 1. Forward pass\n",
|
||||
" y_pred = model(X)\n",
|
||||
"\n",
|
||||
" # 2. Calculate loss\n",
|
||||
" loss = loss_fn(y_pred, y)\n",
|
||||
"\n",
|
||||
" # 3. Optimizer zero grad\n",
|
||||
" optimizer.zero_grad()\n",
|
||||
"\n",
|
||||
" # 4. Loss backward\n",
|
||||
" loss.backward()\n",
|
||||
"\n",
|
||||
" # 5. Optimizer step\n",
|
||||
" optimizer.step()\n",
|
||||
"\n",
|
||||
" # 6. Calculate metrics\n",
|
||||
" train_loss += loss.item()\n",
|
||||
" y_pred_class = torch.softmax(y_pred, dim=1).argmax(dim=1)\n",
|
||||
" # print(f\"y: \\n{y}\\ny_pred_class:{y_pred_class}\")\n",
|
||||
" # print(f\"y argmax: {y_pred.argmax(dim=1)}\")\n",
|
||||
" # print(f\"Equal: {(y_pred_class == y)}\")\n",
|
||||
" train_acc += (y_pred_class == y).sum().item() / len(y_pred)\n",
|
||||
" # print(f\"batch: {batch} train_acc: {train_acc}\")\n",
|
||||
"\n",
|
||||
" # Adjust returned metrics\n",
|
||||
" return train_loss / len(dataloader), train_acc / len(dataloader)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"TK - Now to use the writer, we've got to adjust the `train()` function..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 72,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def train(\n",
|
||||
" model,\n",
|
||||
" train_dataloader,\n",
|
||||
" test_dataloader,\n",
|
||||
" optimizer,\n",
|
||||
" loss_fn=nn.CrossEntropyLoss(),\n",
|
||||
" epochs=5,\n",
|
||||
"):\n",
|
||||
"\n",
|
||||
" results = {\"train_loss\": [], \"train_acc\": [], \"test_loss\": [], \"test_acc\": []}\n",
|
||||
"\n",
|
||||
" for epoch in tqdm(range(epochs)):\n",
|
||||
" train_loss, train_acc = train_step(\n",
|
||||
" model=model,\n",
|
||||
" dataloader=train_dataloader,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" )\n",
|
||||
" test_loss, test_acc = test_step(\n",
|
||||
" model=model, dataloader=test_dataloader, loss_fn=loss_fn\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Print out what's happening\n",
|
||||
" print(\n",
|
||||
" f\"Epoch: {epoch+1} | \"\n",
|
||||
" f\"train_loss: {train_loss:.4f} | \"\n",
|
||||
" f\"train_acc: {train_acc:.4f} | \"\n",
|
||||
" f\"test_loss: {test_loss:.4f} | \"\n",
|
||||
" f\"test_acc: {test_acc:.4f}\"\n",
|
||||
" )\n",
|
||||
"\n",
|
||||
" # Update results\n",
|
||||
" results[\"train_loss\"].append(train_loss)\n",
|
||||
" results[\"train_acc\"].append(train_acc)\n",
|
||||
" results[\"test_loss\"].append(test_loss)\n",
|
||||
" results[\"test_acc\"].append(test_acc)\n",
|
||||
"\n",
|
||||
" # Add results to SummaryWriter\n",
|
||||
" writer.add_scalars(main_tag=\"Loss\", \n",
|
||||
" tag_scalar_dict={\"train_loss\": train_loss,\n",
|
||||
" \"test_loss\": test_loss},\n",
|
||||
" global_step=epoch)\n",
|
||||
" writer.add_scalars(main_tag=\"Accuracy\", \n",
|
||||
" tag_scalar_dict={\"train_acc\": train_acc,\n",
|
||||
" \"test_acc\": test_acc}, \n",
|
||||
" global_step=epoch)\n",
|
||||
" \n",
|
||||
" # Close the writer\n",
|
||||
" writer.close()\n",
|
||||
"\n",
|
||||
" return results"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 73,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 20%|██ | 1/5 [00:05<00:21, 5.27s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 1 | train_loss: 1.9644 | train_acc: 0.4386 | test_loss: 1.5205 | test_acc: 0.7865\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 40%|████ | 2/5 [00:09<00:14, 4.94s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 2 | train_loss: 1.2589 | train_acc: 0.7878 | test_loss: 1.1589 | test_acc: 0.7604\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 60%|██████ | 3/5 [00:14<00:09, 4.72s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 3 | train_loss: 0.8642 | train_acc: 0.8776 | test_loss: 0.9347 | test_acc: 0.7917\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 80%|████████ | 4/5 [00:18<00:04, 4.56s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 4 | train_loss: 0.6827 | train_acc: 0.8856 | test_loss: 0.6637 | test_acc: 0.8750\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|██████████| 5/5 [00:23<00:00, 4.65s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 5 | train_loss: 0.5688 | train_acc: 0.9069 | test_loss: 0.6175 | test_acc: 0.8854\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Train model\n",
|
||||
"# Note: Not using engine.train() since the original script isn't updated\n",
|
||||
"results = train(model=model,\n",
|
||||
" train_dataloader=train_dataloader,\n",
|
||||
" test_dataloader=test_dataloader,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Looks like mixed precision doesn't offer much benefit for smaller feature extraction models..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# # Without mixed precision\n",
|
||||
"# 20%|██ | 1/5 [00:03<00:14, 3.71s/it]Epoch: 1 | train_loss: 0.5229 | train_acc: 0.9054 | test_loss: 0.5776 | test_acc: 0.8542\n",
|
||||
"# 40%|████ | 2/5 [00:07<00:11, 3.74s/it]Epoch: 2 | train_loss: 0.4699 | train_acc: 0.9001 | test_loss: 0.5160 | test_acc: 0.8802\n",
|
||||
"# 60%|██████ | 3/5 [00:10<00:07, 3.63s/it]Epoch: 3 | train_loss: 0.3913 | train_acc: 0.9196 | test_loss: 0.4888 | test_acc: 0.8906\n",
|
||||
"# 80%|████████ | 4/5 [00:14<00:03, 3.61s/it]Epoch: 4 | train_loss: 0.3724 | train_acc: 0.9371 | test_loss: 0.4931 | test_acc: 0.8698\n",
|
||||
"# 100%|██████████| 5/5 [00:18<00:00, 3.61s/it]Epoch: 5 | train_loss: 0.3315 | train_acc: 0.9381 | test_loss: 0.4405 | test_acc: 0.8750\n",
|
||||
"\n",
|
||||
"# # With mixed precision\n",
|
||||
"# 20%|██ | 1/5 [00:04<00:17, 4.40s/it]Epoch: 1 | train_loss: 0.3027 | train_acc: 0.9554 | test_loss: 0.4386 | test_acc: 0.8802\n",
|
||||
"# 40%|████ | 2/5 [00:08<00:13, 4.49s/it]Epoch: 2 | train_loss: 0.2826 | train_acc: 0.9539 | test_loss: 0.4080 | test_acc: 0.8802\n",
|
||||
"# 60%|██████ | 3/5 [00:13<00:08, 4.48s/it]Epoch: 3 | train_loss: 0.2450 | train_acc: 0.9609 | test_loss: 0.4130 | test_acc: 0.8750\n",
|
||||
"# 80%|████████ | 4/5 [00:18<00:04, 4.53s/it]Epoch: 4 | train_loss: 0.2450 | train_acc: 0.9594 | test_loss: 0.4158 | test_acc: 0.8802\n",
|
||||
"# 100%|██████████| 5/5 [00:22<00:00, 4.49s/it]Epoch: 5 | train_loss: 0.2307 | train_acc: 0.9639 | test_loss: 0.4124 | test_acc: 0.8906"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Try mixed precision with larger model\n",
|
||||
"\n",
|
||||
"Now we'll try turn on mixed precision with a larger model (e.g. EffifientNetB0 with all layers tuneable)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 74,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Unfreeze all base layers \n",
|
||||
"for param in model.features.parameters():\n",
|
||||
" param.requires_grad = True\n",
|
||||
"\n",
|
||||
"# for param in model.features.parameters():\n",
|
||||
"# print(param.requires_grad)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 75,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 20%|██ | 1/5 [00:13<00:53, 13.27s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 1 | train_loss: 0.4934 | train_acc: 0.8586 | test_loss: 0.6467 | test_acc: 0.7969\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 40%|████ | 2/5 [00:27<00:42, 14.09s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 2 | train_loss: 0.1750 | train_acc: 0.9628 | test_loss: 1.1806 | test_acc: 0.8385\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 60%|██████ | 3/5 [00:42<00:28, 14.28s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 3 | train_loss: 0.1362 | train_acc: 0.9619 | test_loss: 0.5831 | test_acc: 0.8802\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 80%|████████ | 4/5 [00:57<00:14, 14.47s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 4 | train_loss: 0.1743 | train_acc: 0.9462 | test_loss: 0.5702 | test_acc: 0.8854\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"100%|██████████| 5/5 [01:11<00:00, 14.38s/it]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Epoch: 5 | train_loss: 0.2437 | train_acc: 0.9352 | test_loss: 0.7096 | test_acc: 0.8125\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Train model\n",
|
||||
"# Note: Not using engine.train() since the original script isn't updated\n",
|
||||
"results = train(model=model,\n",
|
||||
" train_dataloader=train_dataloader,\n",
|
||||
" test_dataloader=test_dataloader,\n",
|
||||
" optimizer=optimizer,\n",
|
||||
" loss_fn=loss_fn,\n",
|
||||
" epochs=5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# # Without mixed precision...\n",
|
||||
"# 20%|██ | 1/5 [00:11<00:46, 11.61s/it]Epoch: 1 | train_loss: 0.4507 | train_acc: 0.8648 | test_loss: 1.0603 | test_acc: 0.7604\n",
|
||||
"# 40%|████ | 2/5 [00:24<00:36, 12.21s/it]Epoch: 2 | train_loss: 0.1659 | train_acc: 0.9464 | test_loss: 0.6398 | test_acc: 0.8490\n",
|
||||
"# 60%|██████ | 3/5 [00:36<00:24, 12.38s/it]Epoch: 3 | train_loss: 0.1261 | train_acc: 0.9698 | test_loss: 0.7149 | test_acc: 0.8542\n",
|
||||
"# 80%|████████ | 4/5 [00:49<00:12, 12.53s/it]Epoch: 4 | train_loss: 0.1250 | train_acc: 0.9609 | test_loss: 0.7441 | test_acc: 0.7917\n",
|
||||
"# 100%|██████████| 5/5 [01:02<00:00, 12.42s/it]Epoch: 5 | train_loss: 0.1282 | train_acc: 0.9564 | test_loss: 0.8701 | test_acc: 0.8385\n",
|
||||
"\n",
|
||||
"# # With mixed precision...\n",
|
||||
"# 20%|██ | 1/5 [00:13<00:53, 13.27s/it]Epoch: 1 | train_loss: 0.4934 | train_acc: 0.8586 | test_loss: 0.6467 | test_acc: 0.7969\n",
|
||||
"# 40%|████ | 2/5 [00:27<00:42, 14.09s/it]Epoch: 2 | train_loss: 0.1750 | train_acc: 0.9628 | test_loss: 1.1806 | test_acc: 0.8385\n",
|
||||
"# 60%|██████ | 3/5 [00:42<00:28, 14.28s/it]Epoch: 3 | train_loss: 0.1362 | train_acc: 0.9619 | test_loss: 0.5831 | test_acc: 0.8802\n",
|
||||
"# 80%|████████ | 4/5 [00:57<00:14, 14.47s/it]Epoch: 4 | train_loss: 0.1743 | train_acc: 0.9462 | test_loss: 0.5702 | test_acc: 0.8854\n",
|
||||
"# 100%|██████████| 5/5 [01:11<00:00, 14.38s/it]Epoch: 5 | train_loss: 0.2437 | train_acc: 0.9352 | test_loss: 0.7096 | test_acc: 0.8125"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Checking the PyTorch profiler, it seems that mixed precision utilises some Tensor Cores, however, these aren't large numbers.\n",
|
||||
"\n",
|
||||
"E.g. it uses 9-12% Tensor Cores. Perhaps the slow down when using mixed precision is because the tensors have to get altered and converted when there isn't very many of them. For example only 9-12% of tensors get converted so the speed up gains aren't realised on these tensors because they get cancelled out by the conversion time."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Extensions\n",
|
||||
"* Does changing the data input size to EfficientNetB4 change its results? E.g. input image size of (380, 380) instead of (224, 224)?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
" "
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"interpreter": {
|
||||
"hash": "3fbe1355223f7b2ffc113ba3ade6a2b520cadace5d5ec3e828c83ce02eb221bf"
|
||||
},
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
www.learnpytorch.io
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
+153
@@ -0,0 +1,153 @@
|
||||
# Learn PyTorch for Deep Learning: Zero to Mastery book
|
||||
|
||||
<a href="https://learnpytorch.io">
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/misc-pytorch-course-launch-cover-white-text-black-background.jpg" width=900 alt="pytorch deep learning by zero to mastery cover photo with different sections of the course">
|
||||
</a>
|
||||
|
||||
Welcome to the second best place on the internet to learn PyTorch (the first being the [PyTorch documentation](https://pytorch.org/docs/stable/index.html)).
|
||||
|
||||
This is the online book version of the [Learn PyTorch for Deep Learning: Zero to Mastery course](https://dbourke.link/ZTMPyTorch).
|
||||
|
||||
This course will teach you the foundations of machine learning and deep learning with PyTorch (a machine learning framework written in Python).
|
||||
|
||||
The course is video based. However, the videos are based on the contents of this online book.
|
||||
|
||||
For full code and resources see the [course GitHub](https://github.com/mrdbourke/pytorch-deep-learning).
|
||||
|
||||
Otherwise, you can find more about the course below.
|
||||
|
||||
## Does this course cover PyTorch 2.0?
|
||||
|
||||
Yes. PyTorch 2.0 is an additive release to previous versions of PyTorch.
|
||||
|
||||
This means it adds new features on top of the existing baseline features of PyTorch.
|
||||
|
||||
This course focuses on the baseline features of PyTorch (e.g. you're a beginner wanting to get into deep learning/AI).
|
||||
|
||||
Once you know the fundamentals of PyTorch, PyTorch 2.0 is a quick upgrade, there's a [tutorial on this website](https://www.learnpytorch.io/pytorch_2_intro/) which runs through the new features.
|
||||
|
||||
## Status
|
||||
|
||||
Course [launched on ZTM Academy](https://dbourke.link/ZTMPyTorch)!
|
||||
|
||||
* Last update: April 16 2023
|
||||
* Videos are done for chapters: 00, 01, 02, 03, 04, 05, 06, 07, 08, 09 (all chapters!)
|
||||
* Currently working on: [PyTorch 2.0 Tutorial](https://www.learnpytorch.io/pytorch_2_intro/)
|
||||
* See progress on the course [GitHub Project](https://github.com/users/mrdbourke/projects/1/views/4).
|
||||
|
||||
**Get updates:** Follow the [`pytorch-deep-learning`](https://github.com/mrdbourke/pytorch-deep-learning#log) repo log or [sign up for emails](https://www.mrdbourke.com/newsletter/).
|
||||
|
||||
## Course materials/outline
|
||||
|
||||
* 💻 **Code on GitHub:** All of course materials are available open-source [on GitHub](https://github.com/mrdbourke/pytorch-deep-learning).
|
||||
* 🎥 **First five sections on YouTube:** Learn Pytorch in a day by watching the [first 25-hours of material](https://youtu.be/Z_ikDlimN6A).
|
||||
* 🔬 **Course focus:** code, code, code, experiment, experiment, experiment.
|
||||
* 🏃♂️ **Teaching style:** [https://sive.rs/kimo](https://sive.rs/kimo).
|
||||
* 🤔 **Ask a question:** See the course [GitHub Discussions page](https://github.com/mrdbourke/pytorch-deep-learning/discussions) for existing questions/ask your own.
|
||||
|
||||
| **Section** | **What does it cover?** | **Exercises & Extra-curriculum** | **Slides** |
|
||||
| ----- | ----- | ----- | ----- |
|
||||
| [00 - PyTorch Fundamentals](https://www.learnpytorch.io/00_pytorch_fundamentals/) | Many fundamental PyTorch operations used for deep learning and neural networks. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/00_pytorch_fundamentals/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/00_pytorch_and_deep_learning_fundamentals.pdf) |
|
||||
| [01 - PyTorch Workflow](https://www.learnpytorch.io/01_pytorch_workflow/) | Provides an outline for approaching deep learning problems and building neural networks with PyTorch. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/01_pytorch_workflow/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/01_pytorch_workflow.pdf) |
|
||||
| [02 - PyTorch Neural Network Classification](https://www.learnpytorch.io/02_pytorch_classification/) | Uses the PyTorch workflow from 01 to go through a neural network classification problem. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/02_pytorch_classification/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/02_pytorch_classification.pdf) |
|
||||
| [03 - PyTorch Computer Vision](https://www.learnpytorch.io/03_pytorch_computer_vision/) | Let's see how PyTorch can be used for computer vision problems using the same workflow from 01 & 02. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/03_pytorch_computer_vision/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/03_pytorch_computer_vision.pdf) |
|
||||
| [04 - PyTorch Custom Datasets](https://www.learnpytorch.io/04_pytorch_custom_datasets/) | How do you load a custom dataset into PyTorch? Also we'll be laying the foundations in this notebook for our modular code (covered in 05). | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/04_pytorch_custom_datasets/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/04_pytorch_custom_datasets.pdf) |
|
||||
| [05 - PyTorch Going Modular](https://www.learnpytorch.io/05_pytorch_going_modular/) | PyTorch is designed to be modular, let's turn what we've created into a series of Python scripts (this is how you'll often find PyTorch code in the wild). | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/05_pytorch_going_modular/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/05_pytorch_going_modular.pdf) |
|
||||
| [06 - PyTorch Transfer Learning](https://www.learnpytorch.io/06_pytorch_transfer_learning/) | Let's take a well performing pre-trained model and adjust it to one of our own problems. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/06_pytorch_transfer_learning/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/06_pytorch_transfer_learning.pdf) |
|
||||
| [07 - Milestone Project 1: PyTorch Experiment Tracking](https://www.learnpytorch.io/07_pytorch_experiment_tracking/) | We've built a bunch of models... wouldn't it be good to track how they're all going? | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/07_pytorch_experiment_tracking/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/07_pytorch_experiment_tracking.pdf) |
|
||||
| [08 - Milestone Project 2: PyTorch Paper Replicating](https://www.learnpytorch.io/08_pytorch_paper_replicating/) | PyTorch is the most popular deep learning framework for machine learning research, let's see why by replicating a machine learning paper. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/08_pytorch_paper_replicating/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/08_pytorch_paper_replicating.pdf) |
|
||||
| [09 - Milestone Project 3: Model Deployment](https://www.learnpytorch.io/09_pytorch_model_deployment/) | So we've built a working PyTorch model... how do we get it in the hands of others? Hint: deploy it to the internet. | [Go to exercises & extra-curriculum](https://www.learnpytorch.io/09_pytorch_model_deployment/#exercises) | [Go to slides](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/slides/09_pytorch_model_deployment.pdf) |
|
||||
| [PyTorch Extra Resources](https://www.learnpytorch.io/pytorch_extra_resources/) | This course covers a large amount of PyTorch and deep learning but the field of machine learning is vast, inside here you'll find recommended books and resources for: PyTorch and deep learning, ML engineering, NLP (natural language processing), time series data, where to find datasets and more. | - | - |
|
||||
| [PyTorch Cheatsheet](https://www.learnpytorch.io/pytorch_cheatsheet/) | A very quick overview of some of the main features of PyTorch plus links to various resources where more can be found in the course and in the PyTorch documentation. | - | - |
|
||||
| [Three Most Common Errors in PyTorch](https://www.learnpytorch.io/pytorch_most_common_errors/) | An overview of the three most common errors in PyTorch (shape, device and datatype errors), how they happen and how to fix them. | - | - |
|
||||
| [A Quick PyTorch 2.0 Tutorial](https://www.learnpytorch.io/pytorch_2_intro/) | A fasssssst introduction to PyTorch 2.0, what's new and how to get started along with resources to learn more. | - | - |
|
||||
|
||||
## About this course
|
||||
|
||||
### Who is this course for?
|
||||
|
||||
**You:** Are a beginner in the field of machine learning or deep learning or AI and would like to learn PyTorch.
|
||||
|
||||
**This course:** Teaches you PyTorch and many machine learning, deep learning and AI concepts in a hands-on, code-first way.
|
||||
|
||||
If you already have 1-year+ experience in machine learning, this course may help but it is specifically designed to be beginner-friendly.
|
||||
|
||||
### What are the prerequisites?
|
||||
|
||||
1. 3-6 months coding Python.
|
||||
2. At least one beginner machine learning course (however this might be able to be skipped, resources are linked for many different topics).
|
||||
3. Experience using Jupyter Notebooks or Google Colab (though you can pick this up as we go along).
|
||||
4. A willingness to learn (most important).
|
||||
|
||||
For 1 & 2, I'd recommend the [Zero to Mastery Data Science and Machine Learning Bootcamp](https://dbourke.link/ZTMMLcourse), it'll teach you the fundamentals of machine learning and Python (I'm biased though, I also teach that course).
|
||||
|
||||
### How is the course taught?
|
||||
|
||||
All of the course materials are available for free in an online book at [learnpytorch.io](https://learnpytorch.io). If you like to read, I'd recommend going through the resources there.
|
||||
|
||||
If you prefer to learn via video, the course is also taught in apprenticeship-style format, meaning I write PyTorch code, you write PyTorch code.
|
||||
|
||||
There's a reason the course motto's include *if in doubt, run the code* and *experiment, experiment, experiment!*.
|
||||
|
||||
My whole goal is to help you to do one thing: learn machine learning by writing PyTorch code.
|
||||
|
||||
The code is all written via [Google Colab Notebooks](https://colab.research.google.com) (you could also use Jupyter Notebooks), an incredible free resource to experiment with machine learning.
|
||||
|
||||
### What will I get if I finish the course?
|
||||
|
||||
There's certificates and all that jazz if you go through the videos.
|
||||
|
||||
But certificates are meh.
|
||||
|
||||
You can consider this course a machine learning momentum builder.
|
||||
|
||||
By the end, you'll have written hundreds of lines of PyTorch code.
|
||||
|
||||
And will have been exposed to many of the most important concepts in machine learning.
|
||||
|
||||
So when you go to build your own machine learning projects or inspect a public machine learning project made with PyTorch, it'll feel familiar and if it doesn't, at least you'll know where to look.
|
||||
|
||||
### What will I build in the course?
|
||||
|
||||
We start with the barebone fundamentals of PyTorch and machine learning, so even if you're new to machine learning you'll be caught up to speed.
|
||||
|
||||
Then we’ll explore more advanced areas including PyTorch neural network classification, PyTorch workflows, computer vision, custom datasets, experiment tracking, model deployment, and my personal favourite: transfer learning, a powerful technique for taking what one machine learning model has learned on another problem and applying it to your own!
|
||||
|
||||
Along the way, you’ll build three milestone projects surrounding an overarching project called FoodVision, a neural network computer vision model to classify images of food.
|
||||
|
||||
These milestone projects will help you practice using PyTorch to cover important machine learning concepts and create a portfolio you can show employers and say "here's what I've done".
|
||||
|
||||
### How do I approach this course?
|
||||
|
||||
<img src="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/docs-how-to-approach-this-course-no-header.png" width=900 alt="how to approach this course: 1. code along, 2. explore and experiment, 3. visualize what you don't understand, 4. ask questions, 5. do the exercises, 6. share your work"/>
|
||||
|
||||
As mentioned, the video version of the course is taught apprenticeship style.
|
||||
|
||||
Meaning I write PyTorch code, you write PyTorch code.
|
||||
|
||||
But here's what I recommend:
|
||||
|
||||
1. **Code along (*if in doubt, run the code*)** - Follow along with code and try to write as much of it as you can yourself, keep doing so until you find yourself writing PyTorch code in your subconscious that's when you can stop writing the same code over and over again.
|
||||
2. **Explore and experiment (*experiment, experiment, experiment!*)** - Machine learning (and deep learning) is very experimental. So if you find yourself wanting to try something on your own and ignoring the materials, do it.
|
||||
3. **Visualize what you don't understand (*visualize, visualize, visualize!*)** - Numbers on a page can get confusing. So make things colourful, see what the inputs and outputs of your code looks like.
|
||||
4. **Ask questions** - If you're stuck with something, ask a question, trying searching for it or if nothing comes up, the course [GitHub Discussions page](https://github.com/mrdbourke/pytorch-deep-learning/discussions) will be the place to go.
|
||||
5. **Do the exercises** - Each module of the course comes with a dedicated exercises section. It's important to try these on your own. You will get stuck. But that's the nature of learning something new: everyone gets stuck.
|
||||
6. **Share your work** - If you've learned something cool or even better, made something cool, share it. It could be with the course Discord group or on the course GitHub page or on your own website. The benefits of sharing your work is you get to practice communicating as well as others can help you out if you're not sure of something.
|
||||
|
||||
### Do I need to take things in order?
|
||||
|
||||
The notebooks/chapters build upon each other sequentially but feel free to jump around.
|
||||
|
||||
### How do I get started?
|
||||
|
||||
You can read the materials on any device but this course is best viewed and coded along within a desktop browser.
|
||||
|
||||
The course uses a free tool called Google Colab. If you've got no experience with it, I'd go through the free [Introduction to Google Colab tutorial](https://colab.research.google.com/notebooks/basic_features_overview.ipynb) and then come back here.
|
||||
|
||||
To start:
|
||||
|
||||
1. Click on one of the notebook or section links like "[00. PyTorch Fundamentals](https://www.learnpytorch.io/00_pytorch_fundamentals/)".
|
||||
2. Click the "Open in Colab" button up the top.
|
||||
3. Press SHIFT+Enter a few times and see what happens.
|
||||
|
||||
Happy machine learning!
|
||||
Reference in New Issue
Block a user