![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r8C1WSzsHC7x"
},
"source": [
"# 02. PyTorch Neural Network Classification\n",
"\n",
"## What is a classification problem?\n",
"\n",
"A [classification problem](https://en.wikipedia.org/wiki/Statistical_classification) involves predicting whether something is one thing or another.\n",
"\n",
"For example, you might want to:\n",
"\n",
"| Problem type | What is it? | Example |\n",
"| ----- | ----- | ----- |\n",
"| **Binary classification** | Target can be one of two options, e.g. yes or no | Predict whether or not someone has heart disease based on their health parameters. |\n",
"| **Multi-class classification** | Target can be one of more than two options | Decide whether a photo of is of food, a person or a dog. |\n",
"| **Multi-label classification** | Target can be assigned more than one option | Predict what categories should be assigned to a Wikipedia article (e.g. mathematics, science & philosohpy). |\n",
"\n",
"\n",
"\n",
"Classification, along with regression (predicting a number, covered in [notebook 01](https://www.learnpytorch.io/01_pytorch_workflow/)) is one of the most common types of machine learning problems.\n",
"\n",
"In this notebook, we're going to work through a couple of different classification problems with PyTorch. \n",
"\n",
"In other words, taking a set of inputs and predicting what class those set of inputs belong to.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9wTlTaDKH7Oj"
},
"source": [
"## What we're going to cover\n",
"\n",
"In this notebook we're going to reiterate over the PyTorch workflow we coverd in notebook 01.\n",
"\n",
"\n",
"\n",
"Except instead of trying to predict a straight line (predicting a number, also called a regression problem), we'll be working on a **classification problem**.\n",
"\n",
"Specifically, we're going to cover:\n",
"\n",
"| **Topic** | **Contents** |\n",
"| ----- | ----- |\n",
"| **0. Architecture of a classification neural network** | Neural networks can come in almost any shape or size, but they typically follow a similar floor plan. |\n",
"| **1. Getting binary classification data ready** | Data can be almost anything but to get started we're going to create a simple binary classification dataset. |\n",
"| **2. Building a PyTorch classification model** | Here we'll create a model to learn patterns in the data, we'll also choose a **loss function**, **optimizer** and build a **training loop** specific to classification. | \n",
"| **3. Fitting the model to data (training)** | We've got data and a model, now let's let the model (try to) find patterns in the (**training**) data. |\n",
"| **4. Making predictions and evaluating a model (inference)** | Our model's found patterns in the data, let's compare its findings to the actual (**testing**) data. |\n",
"| **5. Improving a model (from a model perspective)** | We've trained an evaluated a model but it's not working, let's try a few things to improve it. |\n",
"| **6. Non-linearity** | So far our model has only had the ability to model straight lines, what about non-linear (non-straight) lines? |\n",
"| **7. Replicating non-linear functions** | We used **non-linear functions** to help model non-linear data, but what do these look like? |\n",
"| **8. Putting it all together with multi-class classification** | Let's put everything we've done so far for binary classification together with a multi-class classification problem. |\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uxdUc9OfHtgU"
},
"source": [
"## Where can can you get help?\n",
"\n",
"All of the materials for this course [live on GitHub](https://github.com/mrdbourke/pytorch-deep-learning).\n",
"\n",
"And if you run into trouble, you can ask a question on the [Discussions page](https://github.com/mrdbourke/pytorch-deep-learning/discussions) there too.\n",
"\n",
"There's also the [PyTorch developer forums](https://discuss.pytorch.org/), a very helpful place for all things PyTorch. \n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MSLHJiQxH4jU"
},
"source": [
"## 0. Architecture of a classification neural network\n",
"\n",
"Before we get into writing code, let's look at the general architecture of a classification neural network.\n",
"\n",
"| **Hyperparameter** | **Binary Classification** | **Multiclass classification** |\n",
"| --- | --- | --- |\n",
"| **Input layer shape** (`in_features`) | Same as number of features (e.g. 5 for age, sex, height, weight, smoking status in heart disease prediction) | Same as binary classification |\n",
"| **Hidden layer(s)** | Problem specific, minimum = 1, maximum = unlimited | Same as binary classification |\n",
"| **Neurons per hidden layer** | Problem specific, generally 10 to 512 | Same as binary classification |\n",
"| **Output layer shape** (`out_features`) | 1 (one class or the other) | 1 per class (e.g. 3 for food, person or dog photo) |\n",
"| **Hidden layer activation** | Usually [ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html#torch.nn.ReLU) (rectified linear unit) but [can be many others](https://en.wikipedia.org/wiki/Activation_function#Table_of_activation_functions) | Same as binary classification |\n",
"| **Output activation** | [Sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) ([`torch.sigmoid`](https://pytorch.org/docs/stable/generated/torch.sigmoid.html) in PyTorch)| [Softmax](https://en.wikipedia.org/wiki/Softmax_function) ([`torch.softmax`](https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html) in PyTorch) |\n",
"| **Loss function** | [Binary crossentropy](https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_loss_function_and_logistic_regression) ([`torch.nn.BCELoss`](https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html) in PyTorch) | Cross entropy ([`torch.nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) in PyTorch) |\n",
"| **Optimizer** | [SGD](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html) (stochastic gradient descent), [Adam](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html) (see [`torch.optim`](https://pytorch.org/docs/stable/optim.html) for more options) | Same as binary classification |\n",
"\n",
"Of course, this ingredient list of classification neural network components will vary depending on the problem you're working on.\n",
"\n",
"But it's more than enough to get started.\n",
"\n",
"We're going to gets hands-on with this setup throughout this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VwvxFEjKHC71"
},
"source": [
"## 1. Make classification data and get it ready\n",
"\n",
"Let's begin by making some data.\n",
"\n",
"We'll use the [`make_circles()`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_circles.html) method from Scikit-Learn to generate two circles with different coloured dots. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "RGeZvHsyHC72"
},
"outputs": [],
"source": [
"from sklearn.datasets import make_circles\n",
"\n",
"\n",
"# Make 1000 samples \n",
"n_samples = 1000\n",
"\n",
"# Create circles\n",
"X, y = make_circles(n_samples,\n",
" noise=0.03, # a little bit of noise to the dots\n",
" random_state=42) # keep random state so we get the same values"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1FwwzJnQV2jv"
},
"source": [
"Alright, now let's view the first 5 `X` and `y` values."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "oAb8vcIhWEO8",
"outputId": "b7316d88-7733-4981-9b4a-0a98c7cdd829"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"First 5 X features:\n",
"[[ 0.75424625 0.23148074]\n",
" [-0.75615888 0.15325888]\n",
" [-0.81539193 0.17328203]\n",
" [-0.39373073 0.69288277]\n",
" [ 0.44220765 -0.89672343]]\n",
"\n",
"First 5 y labels:\n",
"[1 1 1 1 0]\n"
]
}
],
"source": [
"print(f\"First 5 X features:\\n{X[:5]}\")\n",
"print(f\"\\nFirst 5 y labels:\\n{y[:5]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ATakj2bVWBou"
},
"source": [
"Looks like there's two `X` values per one `y` value. \n",
"\n",
"Let's keep following the data explorer's motto of *visualize, visualize, visualize* and put them into a pandas DataFrame."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 363
},
"id": "XAAqx_8sHC73",
"outputId": "cd6ef4fe-cda3-48db-f2a5-9820660eab14"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" | \n", " | X1 | \n", "X2 | \n", "label | \n", "
|---|---|---|---|
| 0 | \n", "0.754246 | \n", "0.231481 | \n", "1 | \n", "
| 1 | \n", "-0.756159 | \n", "0.153259 | \n", "1 | \n", "
| 2 | \n", "-0.815392 | \n", "0.173282 | \n", "1 | \n", "
| 3 | \n", "-0.393731 | \n", "0.692883 | \n", "1 | \n", "
| 4 | \n", "0.442208 | \n", "-0.896723 | \n", "0 | \n", "
| 5 | \n", "-0.479646 | \n", "0.676435 | \n", "1 | \n", "
| 6 | \n", "-0.013648 | \n", "0.803349 | \n", "1 | \n", "
| 7 | \n", "0.771513 | \n", "0.147760 | \n", "1 | \n", "
| 8 | \n", "-0.169322 | \n", "-0.793456 | \n", "1 | \n", "
| 9 | \n", "-0.121486 | \n", "1.021509 | \n", "0 | \n", "
forward() function\n",
" calculations (model(x_train)).\n",
" loss = loss_fn(y_pred, y_train).optimizer.zero_grad()).requires_grad=True). This is known as backpropagation, hence \"backwards\"\n",
" (loss.backward()).requires_grad=True\n",
" with respect to the loss\n",
" gradients in order to improve them (optimizer.step()).