chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,299 @@
|
||||
# Introduction to classification
|
||||
|
||||
In these four lessons, you will explore a fundamental focus of classic machine learning - _classification_. We will walk through using various classification algorithms with a dataset about all the brilliant cuisines of Asia and India. Hope you're hungry!
|
||||
|
||||
|
||||
|
||||
> Celebrate pan-Asian cuisines in these lessons! Image by [Jen Looper](https://twitter.com/jenlooper)
|
||||
|
||||
Classification is a form of [supervised learning](https://wikipedia.org/wiki/Supervised_learning) that bears a lot in common with regression techniques. If machine learning is all about predicting values or names to things by using datasets, then classification generally falls into two groups: _binary classification_ and _multiclass classification_.
|
||||
|
||||
[](https://youtu.be/eg8DJYwdMyg "Introduction to classification")
|
||||
|
||||
> 🎥 Click the image above for a video: MIT's John Guttag introduces classification
|
||||
|
||||
Remember:
|
||||
|
||||
- **Linear regression** helped you predict relationships between variables and make accurate predictions on where a new datapoint would fall in relationship to that line. So, you could predict _what price a pumpkin would be in September vs. December_, for example.
|
||||
- **Logistic regression** helped you discover "binary categories": at this price point, _is this pumpkin orange or not-orange_?
|
||||
|
||||
Classification uses various algorithms to determine other ways of determining a data point's label or class. Let's work with this cuisine data to see whether, by observing a group of ingredients, we can determine its cuisine of origin.
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
> ### [This lesson is available in R!](./solution/R/lesson_10.html)
|
||||
|
||||
### Introduction
|
||||
|
||||
Classification is one of the fundamental activities of the machine learning researcher and data scientist. From basic classification of a binary value ("is this email spam or not?"), to complex image classification and segmentation using computer vision, it's always useful to be able to sort data into classes and ask questions of it.
|
||||
|
||||
To state the process in a more scientific way, your classification method creates a predictive model that enables you to map the relationship between input variables to output variables.
|
||||
|
||||
|
||||
|
||||
> Binary vs. multiclass problems for classification algorithms to handle. Infographic by [Jen Looper](https://twitter.com/jenlooper)
|
||||
|
||||
Before starting the process of cleaning our data, visualizing it, and prepping it for our ML tasks, let's learn a bit about the various ways machine learning can be leveraged to classify data.
|
||||
|
||||
Derived from [statistics](https://wikipedia.org/wiki/Statistical_classification), classification using classic machine learning uses features, such as `smoker`, `weight`, and `age` to determine _likelihood of developing X disease_. As a supervised learning technique similar to the regression exercises you performed earlier, your data is labeled and the ML algorithms use those labels to classify and predict classes (or 'features') of a dataset and assign them to a group or outcome.
|
||||
|
||||
✅ Take a moment to imagine a dataset about cuisines. What would a multiclass model be able to answer? What would a binary model be able to answer? What if you wanted to determine whether a given cuisine was likely to use fenugreek? What if you wanted to see if, given a present of a grocery bag full of star anise, artichokes, cauliflower, and horseradish, you could create a typical Indian dish?
|
||||
|
||||
[](https://youtu.be/GuTeDbaNoEU "Crazy mystery baskets")
|
||||
|
||||
> 🎥 Click the image above for a video.The whole premise of the show 'Chopped' is the 'mystery basket' where chefs have to make some dish out of a random choice of ingredients. Surely a ML model would have helped!
|
||||
|
||||
## Hello 'classifier'
|
||||
|
||||
The question we want to ask of this cuisine dataset is actually a **multiclass question**, as we have several potential national cuisines to work with. Given a batch of ingredients, which of these many classes will the data fit?
|
||||
|
||||
Scikit-learn offers several different algorithms to use to classify data, depending on the kind of problem you want to solve. In the next two lessons, you'll learn about several of these algorithms.
|
||||
|
||||
## Exercise - clean and balance your data
|
||||
|
||||
The first task at hand, before starting this project, is to clean and **balance** your data to get better results. Start with the blank _notebook.ipynb_ file in the root of this folder.
|
||||
|
||||
The first thing to install is [imblearn](https://imbalanced-learn.org/stable/). This is a Scikit-learn package that will allow you to better balance the data (you will learn more about this task in a minute).
|
||||
|
||||
1. To install `imblearn`, run `pip install`, like so:
|
||||
|
||||
```python
|
||||
pip install imblearn
|
||||
```
|
||||
|
||||
1. Import the packages you need to import your data and visualize it, also import `SMOTE` from `imblearn`.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib as mpl
|
||||
import numpy as np
|
||||
from imblearn.over_sampling import SMOTE
|
||||
```
|
||||
|
||||
Now you are set up to read import the data next.
|
||||
|
||||
1. The next task will be to import the data:
|
||||
|
||||
```python
|
||||
df = pd.read_csv('../data/cuisines.csv')
|
||||
```
|
||||
|
||||
Using `read_csv()` will read the content of the csv file _cusines.csv_ and place it in the variable `df`.
|
||||
|
||||
1. Check the data's shape:
|
||||
|
||||
```python
|
||||
df.head()
|
||||
```
|
||||
|
||||
The first five rows look like this:
|
||||
|
||||
```output
|
||||
| | Unnamed: 0 | cuisine | almond | angelica | anise | anise_seed | apple | apple_brandy | apricot | armagnac | ... | whiskey | white_bread | white_wine | whole_grain_wheat_flour | wine | wood | yam | yeast | yogurt | zucchini |
|
||||
| --- | ---------- | ------- | ------ | -------- | ----- | ---------- | ----- | ------------ | ------- | -------- | --- | ------- | ----------- | ---------- | ----------------------- | ---- | ---- | --- | ----- | ------ | -------- |
|
||||
| 0 | 65 | indian | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 1 | 66 | indian | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 2 | 67 | indian | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 3 | 68 | indian | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
||||
| 4 | 69 | indian | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
|
||||
```
|
||||
|
||||
1. Get info about this data by calling `info()`:
|
||||
|
||||
```python
|
||||
df.info()
|
||||
```
|
||||
|
||||
Your out resembles:
|
||||
|
||||
```output
|
||||
<class 'pandas.core.frame.DataFrame'>
|
||||
RangeIndex: 2448 entries, 0 to 2447
|
||||
Columns: 385 entries, Unnamed: 0 to zucchini
|
||||
dtypes: int64(384), object(1)
|
||||
memory usage: 7.2+ MB
|
||||
```
|
||||
|
||||
## Exercise - learning about cuisines
|
||||
|
||||
Now the work starts to become more interesting. Let's discover the distribution of data, per cuisine
|
||||
|
||||
1. Plot the data as bars by calling `barh()`:
|
||||
|
||||
```python
|
||||
df.cuisine.value_counts().plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
There are a finite number of cuisines, but the distribution of data is uneven. You can fix that! Before doing so, explore a little more.
|
||||
|
||||
1. Find out how much data is available per cuisine and print it out:
|
||||
|
||||
```python
|
||||
thai_df = df[(df.cuisine == "thai")]
|
||||
japanese_df = df[(df.cuisine == "japanese")]
|
||||
chinese_df = df[(df.cuisine == "chinese")]
|
||||
indian_df = df[(df.cuisine == "indian")]
|
||||
korean_df = df[(df.cuisine == "korean")]
|
||||
|
||||
print(f'thai df: {thai_df.shape}')
|
||||
print(f'japanese df: {japanese_df.shape}')
|
||||
print(f'chinese df: {chinese_df.shape}')
|
||||
print(f'indian df: {indian_df.shape}')
|
||||
print(f'korean df: {korean_df.shape}')
|
||||
```
|
||||
|
||||
the output looks like so:
|
||||
|
||||
```output
|
||||
thai df: (289, 385)
|
||||
japanese df: (320, 385)
|
||||
chinese df: (442, 385)
|
||||
indian df: (598, 385)
|
||||
korean df: (799, 385)
|
||||
```
|
||||
|
||||
## Discovering ingredients
|
||||
|
||||
Now you can dig deeper into the data and learn what are the typical ingredients per cuisine. You should clean out recurrent data that creates confusion between cuisines, so let's learn about this problem.
|
||||
|
||||
1. Create a function `create_ingredient()` in Python to create an ingredient dataframe. This function will start by dropping an unhelpful column and sort through ingredients by their count:
|
||||
|
||||
```python
|
||||
def create_ingredient_df(df):
|
||||
ingredient_df = df.T.drop(['cuisine','Unnamed: 0']).sum(axis=1).to_frame('value')
|
||||
ingredient_df = ingredient_df[(ingredient_df.T != 0).any()]
|
||||
ingredient_df = ingredient_df.sort_values(by='value', ascending=False,
|
||||
inplace=False)
|
||||
return ingredient_df
|
||||
```
|
||||
|
||||
Now you can use that function to get an idea of top ten most popular ingredients by cuisine.
|
||||
|
||||
1. Call `create_ingredient()` and plot it calling `barh()`:
|
||||
|
||||
```python
|
||||
thai_ingredient_df = create_ingredient_df(thai_df)
|
||||
thai_ingredient_df.head(10).plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
1. Do the same for the japanese data:
|
||||
|
||||
```python
|
||||
japanese_ingredient_df = create_ingredient_df(japanese_df)
|
||||
japanese_ingredient_df.head(10).plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
1. Now for the chinese ingredients:
|
||||
|
||||
```python
|
||||
chinese_ingredient_df = create_ingredient_df(chinese_df)
|
||||
chinese_ingredient_df.head(10).plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
1. Plot the indian ingredients:
|
||||
|
||||
```python
|
||||
indian_ingredient_df = create_ingredient_df(indian_df)
|
||||
indian_ingredient_df.head(10).plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
1. Finally, plot the korean ingredients:
|
||||
|
||||
```python
|
||||
korean_ingredient_df = create_ingredient_df(korean_df)
|
||||
korean_ingredient_df.head(10).plot.barh()
|
||||
```
|
||||
|
||||
|
||||
|
||||
1. Now, drop the most common ingredients that create confusion between distinct cuisines, by calling `drop()`:
|
||||
|
||||
Everyone loves rice, garlic and ginger!
|
||||
|
||||
```python
|
||||
feature_df= df.drop(['cuisine','Unnamed: 0','rice','garlic','ginger'], axis=1)
|
||||
labels_df = df.cuisine #.unique()
|
||||
feature_df.head()
|
||||
```
|
||||
|
||||
## Balance the dataset
|
||||
|
||||
Now that you have cleaned the data, use [SMOTE](https://imbalanced-learn.org/dev/references/generated/imblearn.over_sampling.SMOTE.html) - "Synthetic Minority Over-sampling Technique" - to balance it.
|
||||
|
||||
1. Call `fit_resample()`, this strategy generates new samples by interpolation.
|
||||
|
||||
```python
|
||||
oversample = SMOTE()
|
||||
transformed_feature_df, transformed_label_df = oversample.fit_resample(feature_df, labels_df)
|
||||
```
|
||||
|
||||
By balancing your data, you'll have better results when classifying it. Think about a binary classification. If most of your data is one class, a ML model is going to predict that class more frequently, just because there is more data for it. Balancing the data takes any skewed data and helps remove this imbalance.
|
||||
|
||||
1. Now you can check the numbers of labels per ingredient:
|
||||
|
||||
```python
|
||||
print(f'new label count: {transformed_label_df.value_counts()}')
|
||||
print(f'old label count: {df.cuisine.value_counts()}')
|
||||
```
|
||||
|
||||
Your output looks like so:
|
||||
|
||||
```output
|
||||
new label count: korean 799
|
||||
chinese 799
|
||||
indian 799
|
||||
japanese 799
|
||||
thai 799
|
||||
Name: cuisine, dtype: int64
|
||||
old label count: korean 799
|
||||
indian 598
|
||||
chinese 442
|
||||
japanese 320
|
||||
thai 289
|
||||
Name: cuisine, dtype: int64
|
||||
```
|
||||
|
||||
The data is nice and clean, balanced, and very delicious!
|
||||
|
||||
1. The last step is to save your balanced data, including labels and features, into a new dataframe that can be exported into a file:
|
||||
|
||||
```python
|
||||
transformed_df = pd.concat([transformed_label_df,transformed_feature_df],axis=1, join='outer')
|
||||
```
|
||||
|
||||
1. You can take one more look at the data using `transformed_df.head()` and `transformed_df.info()`. Save a copy of this data for use in future lessons:
|
||||
|
||||
```python
|
||||
transformed_df.head()
|
||||
transformed_df.info()
|
||||
transformed_df.to_csv("../data/cleaned_cuisines.csv")
|
||||
```
|
||||
|
||||
This fresh CSV can now be found in the root data folder.
|
||||
|
||||
---
|
||||
|
||||
## 🚀Challenge
|
||||
|
||||
This curriculum contains several interesting datasets. Dig through the `data` folders and see if any contain datasets that would be appropriate for binary or multi-class classification? What questions would you ask of this dataset?
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
Explore SMOTE's API. What use cases is it best used for? What problems does it solve?
|
||||
|
||||
## Assignment
|
||||
|
||||
[Explore classification methods](assignment.md)
|
||||
@@ -0,0 +1,11 @@
|
||||
# Explore classification methods
|
||||
|
||||
## Instructions
|
||||
|
||||
In [Scikit-learn documentation](https://scikit-learn.org/stable/supervised_learning.html) you'll find a large list of ways to classify data. Do a little scavenger hunt in these docs: your goals is to look for classification methods and match a dataset in this curriculum, a question you can ask of it, and a technique of classification. Create a spreadsheet or table in a .doc file and explain how the dataset would work with the classification algorithm.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| | a document is presented overviewing 5 algorithms alongside a classification technique. The overview is well-explained and detailed. | a document is presented overviewing 3 algorithms alongside a classification technique. The overview is well-explained and detailed. | a document is presented overviewing fewer than three algorithms alongside a classification technique and the overview is neither well-explained nor detailed. |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 240 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 624 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.6 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 393 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.7 KiB |
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": 3
|
||||
},
|
||||
"orig_nbformat": 2
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2,
|
||||
"cells": [
|
||||
{
|
||||
"source": [
|
||||
"# Delicious Asian and Indian Cuisines "
|
||||
],
|
||||
"cell_type": "markdown",
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
This is a temporary placeholder
|
||||
@@ -0,0 +1,721 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "lesson_10-R.ipynb",
|
||||
"provenance": [],
|
||||
"collapsed_sections": []
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "ir",
|
||||
"display_name": "R"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "R"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Build a classification model: Delicious Asian and Indian Cuisines"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ItETB4tSFprR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Introduction to classification: Clean, prep, and visualize your data\n",
|
||||
"\n",
|
||||
"In these four lessons, you will explore a fundamental focus of classic machine learning - *classification*. We will walk through using various classification algorithms with a dataset about all the brilliant cuisines of Asia and India. Hope you're hungry!\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/pinch.png\"\n",
|
||||
" width=\"600\"/>\n",
|
||||
" <figcaption>Celebrate pan-Asian cuisines in these lessons! Image by Jen Looper</figcaption>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<!---->\n",
|
||||
"\n",
|
||||
"Classification is a form of [supervised learning](https://wikipedia.org/wiki/Supervised_learning) that bears a lot in common with regression techniques. In classification, you train a model to predict which `category` an item belongs to. If machine learning is all about predicting values or names to things by using datasets, then classification generally falls into two groups: *binary classification* and *multiclass classification*.\n",
|
||||
"\n",
|
||||
"Remember:\n",
|
||||
"\n",
|
||||
"- **Linear regression** helped you predict relationships between variables and make accurate predictions on where a new datapoint would fall in relationship to that line. So, you could predict a numeric values such as *what price a pumpkin would be in September vs. December*, for example.\n",
|
||||
"\n",
|
||||
"- **Logistic regression** helped you discover \"binary categories\": at this price point, *is this pumpkin orange or not-orange*?\n",
|
||||
"\n",
|
||||
"Classification uses various algorithms to determine other ways of determining a data point's label or class. Let's work with this cuisine data to see whether, by observing a group of ingredients, we can determine its cuisine of origin.\n",
|
||||
"\n",
|
||||
"### [**Pre-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/19/)\n",
|
||||
"\n",
|
||||
"### **Introduction**\n",
|
||||
"\n",
|
||||
"Classification is one of the fundamental activities of the machine learning researcher and data scientist. From basic classification of a binary value (\"is this email spam or not?\"), to complex image classification and segmentation using computer vision, it's always useful to be able to sort data into classes and ask questions of it.\n",
|
||||
"\n",
|
||||
"To state the process in a more scientific way, your classification method creates a predictive model that enables you to map the relationship between input variables to output variables.\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/binary-multiclass.png\"\n",
|
||||
" width=\"600\"/>\n",
|
||||
" <figcaption>Binary vs. multiclass problems for classification algorithms to handle. Infographic by Jen Looper</figcaption>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Before starting the process of cleaning our data, visualizing it, and prepping it for our ML tasks, let's learn a bit about the various ways machine learning can be leveraged to classify data.\n",
|
||||
"\n",
|
||||
"Derived from [statistics](https://wikipedia.org/wiki/Statistical_classification), classification using classic machine learning uses features, such as `smoker`, `weight`, and `age` to determine *likelihood of developing X disease*. As a supervised learning technique similar to the regression exercises you performed earlier, your data is labeled and the ML algorithms use those labels to classify and predict classes (or 'features') of a dataset and assign them to a group or outcome.\n",
|
||||
"\n",
|
||||
"✅ Take a moment to imagine a dataset about cuisines. What would a multiclass model be able to answer? What would a binary model be able to answer? What if you wanted to determine whether a given cuisine was likely to use fenugreek? What if you wanted to see if, given a present of a grocery bag full of star anise, artichokes, cauliflower, and horseradish, you could create a typical Indian dish?\n",
|
||||
"\n",
|
||||
"### **Hello 'classifier'**\n",
|
||||
"\n",
|
||||
"The question we want to ask of this cuisine dataset is actually a **multiclass question**, as we have several potential national cuisines to work with. Given a batch of ingredients, which of these many classes will the data fit?\n",
|
||||
"\n",
|
||||
"Tidymodels offers several different algorithms to use to classify data, depending on the kind of problem you want to solve. In the next two lessons, you'll learn about several of these algorithms.\n",
|
||||
"\n",
|
||||
"#### **Prerequisite**\n",
|
||||
"\n",
|
||||
"For this lesson, we'll require the following packages to clean, prep and visualize our data:\n",
|
||||
"\n",
|
||||
"- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!\n",
|
||||
"\n",
|
||||
"- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.\n",
|
||||
"\n",
|
||||
"- `DataExplorer`: The [DataExplorer package](https://cran.r-project.org/web/packages/DataExplorer/vignettes/dataexplorer-intro.html) is meant to simplify and automate EDA process and report generation.\n",
|
||||
"\n",
|
||||
"- `themis`: The [themis package](https://themis.tidymodels.org/) provides Extra Recipes Steps for Dealing with Unbalanced Data.\n",
|
||||
"\n",
|
||||
"You can have them installed as:\n",
|
||||
"\n",
|
||||
"`install.packages(c(\"tidyverse\", \"tidymodels\", \"DataExplorer\", \"here\"))`\n",
|
||||
"\n",
|
||||
"Alternately, the script below checks whether you have the packages required to complete this module and installs them for you in case they are missing."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ri5bQxZ-Fz_0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"suppressWarnings(if (!require(\"pacman\"))install.packages(\"pacman\"))\r\n",
|
||||
"\r\n",
|
||||
"pacman::p_load(tidyverse, tidymodels, DataExplorer, themis, here)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "KIPxa4elGAPI"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"We'll later load these awesome packages and make them available in our current R session. (This is for mere illustration, `pacman::p_load()` already did that for you)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YkKAxOJvGD4C"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Exercise - clean and balance your data\n",
|
||||
"\n",
|
||||
"The first task at hand, before starting this project, is to clean and **balance** your data to get better results\n",
|
||||
"\n",
|
||||
"Let's meet the data!🕵️"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "PFkQDlk0GN5O"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Import data\r\n",
|
||||
"df <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/4-Classification/data/cuisines.csv\")\r\n",
|
||||
"\r\n",
|
||||
"# View the first 5 rows\r\n",
|
||||
"df %>% \r\n",
|
||||
" slice_head(n = 5)\r\n"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "Qccw7okxGT0S"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Interesting! From the looks of it, the first column is a kind of `id` column. Let's get a little more information about the data."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XrWnlgSrGVmR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Basic information about the data\r\n",
|
||||
"df %>%\r\n",
|
||||
" introduce()\r\n",
|
||||
"\r\n",
|
||||
"# Visualize basic information above\r\n",
|
||||
"df %>% \r\n",
|
||||
" plot_intro(ggtheme = theme_light())"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "4UcGmxRxGieA"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"From the output, we can immediately see that we have `2448` rows and `385` columns and `0` missing values. We also have 1 discrete column, *cuisine*.\n",
|
||||
"\n",
|
||||
"## Exercise - learning about cuisines\n",
|
||||
"\n",
|
||||
"Now the work starts to become more interesting. Let's discover the distribution of data, per cuisine."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "AaPubl__GmH5"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Count observations per cuisine\r\n",
|
||||
"df %>% \r\n",
|
||||
" count(cuisine) %>% \r\n",
|
||||
" arrange(n)\r\n",
|
||||
"\r\n",
|
||||
"# Plot the distribution\r\n",
|
||||
"theme_set(theme_light())\r\n",
|
||||
"df %>% \r\n",
|
||||
" count(cuisine) %>% \r\n",
|
||||
" ggplot(mapping = aes(x = n, y = reorder(cuisine, -n))) +\r\n",
|
||||
" geom_col(fill = \"midnightblue\", alpha = 0.7) +\r\n",
|
||||
" ylab(\"cuisine\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "FRsBVy5eGrrv"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"There are a finite number of cuisines, but the distribution of data is uneven. You can fix that! Before doing so, explore a little more.\r\n",
|
||||
"\r\n",
|
||||
"Next, let's assign each cuisine into its individual tibble and find out how much data is available (rows, columns) per cuisine.\r\n",
|
||||
"\r\n",
|
||||
"> A [tibble](https://tibble.tidyverse.org/) is a modern data frame.\r\n",
|
||||
"\r\n",
|
||||
"<p >\r\n",
|
||||
" <img src=\"../../images/dplyr_filter.jpg\"\r\n",
|
||||
" width=\"600\"/>\r\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\r\n",
|
||||
"\r\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "vVvyDb1kG2in"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Create individual tibble for the cuisines\r\n",
|
||||
"thai_df <- df %>% \r\n",
|
||||
" filter(cuisine == \"thai\")\r\n",
|
||||
"japanese_df <- df %>% \r\n",
|
||||
" filter(cuisine == \"japanese\")\r\n",
|
||||
"chinese_df <- df %>% \r\n",
|
||||
" filter(cuisine == \"chinese\")\r\n",
|
||||
"indian_df <- df %>% \r\n",
|
||||
" filter(cuisine == \"indian\")\r\n",
|
||||
"korean_df <- df %>% \r\n",
|
||||
" filter(cuisine == \"korean\")\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"# Find out how much data is available per cuisine\r\n",
|
||||
"cat(\" thai df:\", dim(thai_df), \"\\n\",\r\n",
|
||||
" \"japanese df:\", dim(japanese_df), \"\\n\",\r\n",
|
||||
" \"chinese_df:\", dim(chinese_df), \"\\n\",\r\n",
|
||||
" \"indian_df:\", dim(indian_df), \"\\n\",\r\n",
|
||||
" \"korean_df:\", dim(korean_df))"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "0TvXUxD3G8Bk"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Perfect!😋\n",
|
||||
"\n",
|
||||
"## **Exercise - Discovering top ingredients by cuisine using dplyr**\n",
|
||||
"\n",
|
||||
"Now you can dig deeper into the data and learn what are the typical ingredients per cuisine. You should clean out recurrent data that creates confusion between cuisines, so let's learn about this problem.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Create a function `create_ingredient()` in R that returns an ingredient dataframe. This function will start by dropping an unhelpful column and sort through ingredients by their count.\n",
|
||||
"\n",
|
||||
"The basic structure of a function in R is:\n",
|
||||
"\n",
|
||||
"`myFunction <- function(arglist){`\n",
|
||||
"\n",
|
||||
"**`...`**\n",
|
||||
"\n",
|
||||
"**`return`**`(value)`\n",
|
||||
"\n",
|
||||
"`}`\n",
|
||||
"\n",
|
||||
"A tidy introduction to R functions can be found [here](https://skirmer.github.io/presentations/functions_with_r.html#1).\n",
|
||||
"\n",
|
||||
"Let's get right into it! We'll make use of [dplyr verbs](https://dplyr.tidyverse.org/) which we have been learning in our previous lessons. As a recap:\n",
|
||||
"\n",
|
||||
"- `dplyr::select()`: help you pick which **columns** to keep or exclude.\n",
|
||||
"\n",
|
||||
"- `dplyr::pivot_longer()`: helps you to \"lengthen\" data, increasing the number of rows and decreasing the number of columns.\n",
|
||||
"\n",
|
||||
"- `dplyr::group_by()` and `dplyr::summarise()`: helps you to find find summary statistics for different groups, and put them in a nice table.\n",
|
||||
"\n",
|
||||
"- `dplyr::filter()`: creates a subset of the data only containing rows that satisfy your conditions.\n",
|
||||
"\n",
|
||||
"- `dplyr::mutate()`: helps you to create or modify columns.\n",
|
||||
"\n",
|
||||
"Check out this [*art*-filled learnr tutorial](https://allisonhorst.shinyapps.io/dplyr-learnr/#section-welcome) by Allison Horst, that introduces some useful data wrangling functions in dplyr *(part of the Tidyverse)*"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "K3RF5bSCHC76"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Creates a functions that returns the top ingredients by class\r\n",
|
||||
"\r\n",
|
||||
"create_ingredient <- function(df){\r\n",
|
||||
" \r\n",
|
||||
" # Drop the id column which is the first colum\r\n",
|
||||
" ingredient_df = df %>% select(-1) %>% \r\n",
|
||||
" # Transpose data to a long format\r\n",
|
||||
" pivot_longer(!cuisine, names_to = \"ingredients\", values_to = \"count\") %>% \r\n",
|
||||
" # Find the top most ingredients for a particular cuisine\r\n",
|
||||
" group_by(ingredients) %>% \r\n",
|
||||
" summarise(n_instances = sum(count)) %>% \r\n",
|
||||
" filter(n_instances != 0) %>% \r\n",
|
||||
" # Arrange by descending order\r\n",
|
||||
" arrange(desc(n_instances)) %>% \r\n",
|
||||
" mutate(ingredients = factor(ingredients) %>% fct_inorder())\r\n",
|
||||
" \r\n",
|
||||
" \r\n",
|
||||
" return(ingredient_df)\r\n",
|
||||
"} # End of function"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "uB_0JR82HTPa"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now we can use the function to get an idea of top ten most popular ingredient by cuisine. Let's take it out for a spin with `thai_df`\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "h9794WF8HWmc"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Call create_ingredient and display popular ingredients\r\n",
|
||||
"thai_ingredient_df <- create_ingredient(df = thai_df)\r\n",
|
||||
"\r\n",
|
||||
"thai_ingredient_df %>% \r\n",
|
||||
" slice_head(n = 10)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "agQ-1HrcHaEA"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"In the previous section, we used `geom_col()`, let's see how you can use `geom_bar` too, to create bar charts. Use `?geom_bar` for further reading."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "kHu9ffGjHdcX"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Make a bar chart for popular thai cuisines\r\n",
|
||||
"thai_ingredient_df %>% \r\n",
|
||||
" slice_head(n = 10) %>% \r\n",
|
||||
" ggplot(aes(x = n_instances, y = ingredients)) +\r\n",
|
||||
" geom_bar(stat = \"identity\", width = 0.5, fill = \"steelblue\") +\r\n",
|
||||
" xlab(\"\") + ylab(\"\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "fb3Bx_3DHj6e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's do the same for the Japanese data"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "RHP_xgdkHnvM"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Get popular ingredients for Japanese cuisines and make bar chart\r\n",
|
||||
"create_ingredient(df = japanese_df) %>% \r\n",
|
||||
" slice_head(n = 10) %>%\r\n",
|
||||
" ggplot(aes(x = n_instances, y = ingredients)) +\r\n",
|
||||
" geom_bar(stat = \"identity\", width = 0.5, fill = \"darkorange\", alpha = 0.8) +\r\n",
|
||||
" xlab(\"\") + ylab(\"\")\r\n"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "019v8F0XHrRU"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"What about the Chinese cuisines?\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "iIGM7vO8Hu3v"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Get popular ingredients for Chinese cuisines and make bar chart\r\n",
|
||||
"create_ingredient(df = chinese_df) %>% \r\n",
|
||||
" slice_head(n = 10) %>%\r\n",
|
||||
" ggplot(aes(x = n_instances, y = ingredients)) +\r\n",
|
||||
" geom_bar(stat = \"identity\", width = 0.5, fill = \"cyan4\", alpha = 0.8) +\r\n",
|
||||
" xlab(\"\") + ylab(\"\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "lHd9_gd2HyzU"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's take a look at the Indian cuisines 🌶️."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "ir8qyQbNH1c7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Get popular ingredients for Indian cuisines and make bar chart\r\n",
|
||||
"create_ingredient(df = indian_df) %>% \r\n",
|
||||
" slice_head(n = 10) %>%\r\n",
|
||||
" ggplot(aes(x = n_instances, y = ingredients)) +\r\n",
|
||||
" geom_bar(stat = \"identity\", width = 0.5, fill = \"#041E42FF\", alpha = 0.8) +\r\n",
|
||||
" xlab(\"\") + ylab(\"\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "ApukQtKjH5FO"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Finally, plot the Korean ingredients."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "qv30cwY1H-FM"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Get popular ingredients for Korean cuisines and make bar chart\r\n",
|
||||
"create_ingredient(df = korean_df) %>% \r\n",
|
||||
" slice_head(n = 10) %>%\r\n",
|
||||
" ggplot(aes(x = n_instances, y = ingredients)) +\r\n",
|
||||
" geom_bar(stat = \"identity\", width = 0.5, fill = \"#852419FF\", alpha = 0.8) +\r\n",
|
||||
" xlab(\"\") + ylab(\"\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "lumgk9cHIBie"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"From the data visualizations, we can now drop the most common ingredients that create confusion between distinct cuisines, using `dplyr::select()`.\n",
|
||||
"\n",
|
||||
"Everyone loves rice, garlic and ginger!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "iO4veMXuIEta"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Drop id column, rice, garlic and ginger from our original data set\r\n",
|
||||
"df_select <- df %>% \r\n",
|
||||
" select(-c(1, rice, garlic, ginger))\r\n",
|
||||
"\r\n",
|
||||
"# Display new data set\r\n",
|
||||
"df_select %>% \r\n",
|
||||
" slice_head(n = 5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "iHJPiG6rIUcK"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Preprocessing data using recipes 👩🍳👨🍳 - Dealing with imbalanced data ⚖️\r\n",
|
||||
"\r\n",
|
||||
"<p >\r\n",
|
||||
" <img src=\"../../images/recipes.png\"\r\n",
|
||||
" width=\"600\"/>\r\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\r\n",
|
||||
"\r\n",
|
||||
"Given that this lesson is about cuisines, we have to put `recipes` into context .\r\n",
|
||||
"\r\n",
|
||||
"Tidymodels provides yet another neat package: `recipes`- a package for preprocessing data.\r\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "kkFd-JxdIaL6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's take a look at the distribution of our cuisines again.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "6l2ubtTPJAhY"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Distribution of cuisines\r\n",
|
||||
"old_label_count <- df_select %>% \r\n",
|
||||
" count(cuisine) %>% \r\n",
|
||||
" arrange(desc(n))\r\n",
|
||||
"\r\n",
|
||||
"old_label_count"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "1e-E9cb7JDVi"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"\n",
|
||||
"As you can see, there is quite an unequal distribution in the number of cuisines. Korean cuisines are almost 3 times Thai cuisines. Imbalanced data often has negative effects on the model performance. Think about a binary classification. If most of your data is one class, a ML model is going to predict that class more frequently, just because there is more data for it. Balancing the data takes any skewed data and helps remove this imbalance. Many models perform best when the number of observations is equal and, thus, tend to struggle with unbalanced data.\n",
|
||||
"\n",
|
||||
"There are majorly two ways of dealing with imbalanced data sets:\n",
|
||||
"\n",
|
||||
"- adding observations to the minority class: `Over-sampling` e.g using a SMOTE algorithm\n",
|
||||
"\n",
|
||||
"- removing observations from majority class: `Under-sampling`\n",
|
||||
"\n",
|
||||
"Let's now demonstrate how to deal with imbalanced data sets using a `recipe`. A recipe can be thought of as a blueprint that describes what steps should be applied to a data set in order to get it ready for data analysis."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "soAw6826JKx9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Load themis package for dealing with imbalanced data\r\n",
|
||||
"library(themis)\r\n",
|
||||
"\r\n",
|
||||
"# Create a recipe for preprocessing data\r\n",
|
||||
"cuisines_recipe <- recipe(cuisine ~ ., data = df_select) %>% \r\n",
|
||||
" step_smote(cuisine)\r\n",
|
||||
"\r\n",
|
||||
"cuisines_recipe"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "HS41brUIJVJy"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's break down our preprocessing steps.\n",
|
||||
"\n",
|
||||
"- The call to `recipe()` with a formula tells the recipe the *roles* of the variables using `df_select` data as the reference. For instance the `cuisine` column has been assigned an `outcome` role while the rest of the columns have been assigned a `predictor` role.\n",
|
||||
"\n",
|
||||
"- [`step_smote(cuisine)`](https://themis.tidymodels.org/reference/step_smote.html) creates a *specification* of a recipe step that synthetically generates new examples of the minority class using nearest neighbors of these cases.\n",
|
||||
"\n",
|
||||
"Now, if we wanted to see the preprocessed data, we'd have to [**`prep()`**](https://recipes.tidymodels.org/reference/prep.html) and [**`bake()`**](https://recipes.tidymodels.org/reference/bake.html) our recipe.\n",
|
||||
"\n",
|
||||
"`prep()`: estimates the required parameters from a training set that can be later applied to other data sets.\n",
|
||||
"\n",
|
||||
"`bake()`: takes a prepped recipe and applies the operations to any data set.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Yb-7t7XcJaC8"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Prep and bake the recipe\r\n",
|
||||
"preprocessed_df <- cuisines_recipe %>% \r\n",
|
||||
" prep() %>% \r\n",
|
||||
" bake(new_data = NULL) %>% \r\n",
|
||||
" relocate(cuisine)\r\n",
|
||||
"\r\n",
|
||||
"# Display data\r\n",
|
||||
"preprocessed_df %>% \r\n",
|
||||
" slice_head(n = 5)\r\n",
|
||||
"\r\n",
|
||||
"# Quick summary stats\r\n",
|
||||
"preprocessed_df %>% \r\n",
|
||||
" introduce()"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "9QhSgdpxJl44"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Let's now check the distribution of our cuisines and compare them with the imbalanced data."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "dmidELh_LdV7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Distribution of cuisines\r\n",
|
||||
"new_label_count <- preprocessed_df %>% \r\n",
|
||||
" count(cuisine) %>% \r\n",
|
||||
" arrange(desc(n))\r\n",
|
||||
"\r\n",
|
||||
"list(new_label_count = new_label_count,\r\n",
|
||||
" old_label_count = old_label_count)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "aSh23klBLwDz"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Yum! The data is nice and clean, balanced, and very delicious 😋!\n",
|
||||
"\n",
|
||||
"> Normally, a recipe is usually used as a preprocessor for modelling where it defines what steps should be applied to a data set in order to get it ready for modelling. In that case, a `workflow()` is typically used (as we have already seen in our previous lessons) instead of manually estimating a recipe\n",
|
||||
">\n",
|
||||
"> As such, you don't typically need to **`prep()`** and **`bake()`** recipes when you use tidymodels, but they are helpful functions to have in your toolkit for confirming that recipes are doing what you expect like in our case.\n",
|
||||
">\n",
|
||||
"> When you **`bake()`** a prepped recipe with **`new_data = NULL`**, you get the data that you provided when defining the recipe back, but having undergone the preprocessing steps.\n",
|
||||
"\n",
|
||||
"Let's now save a copy of this data for use in future lessons:\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "HEu80HZ8L7ae"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Save preprocessed data\r\n",
|
||||
"write_csv(preprocessed_df, \"../../../data/cleaned_cuisines_R.csv\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "cBmCbIgrMOI6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"This fresh CSV can now be found in the root data folder.\r\n",
|
||||
"\r\n",
|
||||
"**🚀Challenge**\r\n",
|
||||
"\r\n",
|
||||
"This curriculum contains several interesting datasets. Dig through the `data` folders and see if any contain datasets that would be appropriate for binary or multi-class classification? What questions would you ask of this dataset?\r\n",
|
||||
"\r\n",
|
||||
"## [**Post-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/20/)\r\n",
|
||||
"\r\n",
|
||||
"## **Review & Self Study**\r\n",
|
||||
"\r\n",
|
||||
"- Check out [package themis](https://github.com/tidymodels/themis). What other techniques could we use to deal with imbalanced data?\r\n",
|
||||
"\r\n",
|
||||
"- Tidy models [reference website](https://www.tidymodels.org/start/).\r\n",
|
||||
"\r\n",
|
||||
"- H. Wickham and G. Grolemund, [*R for Data Science: Visualize, Model, Transform, Tidy, and Import Data*](https://r4ds.had.co.nz/).\r\n",
|
||||
"\r\n",
|
||||
"#### THANK YOU TO:\r\n",
|
||||
"\r\n",
|
||||
"[`Allison Horst`](https://twitter.com/allison_horst/) for creating the amazing illustrations that make R more welcoming and engaging. Find more illustrations at her [gallery](https://www.google.com/url?q=https://github.com/allisonhorst/stats-illustrations&sa=D&source=editors&ust=1626380772530000&usg=AOvVaw3zcfyCizFQZpkSLzxiiQEM).\r\n",
|
||||
"\r\n",
|
||||
"[Cassie Breviu](https://www.twitter.com/cassieview) and [Jen Looper](https://www.twitter.com/jenlooper) for creating the original Python version of this module ♥️\r\n",
|
||||
"\r\n",
|
||||
"<p >\r\n",
|
||||
" <img src=\"../../images/r_learners_sm.jpeg\"\r\n",
|
||||
" width=\"600\"/>\r\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\r\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "WQs5621pMGwf"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,422 @@
|
||||
---
|
||||
title: 'Build a classification model: Delicious Asian and Indian Cuisines'
|
||||
output:
|
||||
html_document:
|
||||
df_print: paged
|
||||
theme: flatly
|
||||
highlight: breezedark
|
||||
toc: yes
|
||||
toc_float: yes
|
||||
code_download: yes
|
||||
---
|
||||
|
||||
## Introduction to classification: Clean, prep, and visualize your data
|
||||
|
||||
In these four lessons, you will explore a fundamental focus of classic machine learning - *classification*. We will walk through using various classification algorithms with a dataset about all the brilliant cuisines of Asia and India. Hope you're hungry!
|
||||
|
||||
|
||||
|
||||
Classification is a form of [supervised learning](https://wikipedia.org/wiki/Supervised_learning) that bears a lot in common with regression techniques. In classification, you train a model to predict which `category` an item belongs to. If machine learning is all about predicting values or names to things by using datasets, then classification generally falls into two groups: *binary classification* and *multiclass classification*.
|
||||
|
||||
Remember:
|
||||
|
||||
- **Linear regression** helped you predict relationships between variables and make accurate predictions on where a new datapoint would fall in relationship to that line. So, you could predict a numeric values such as *what price a pumpkin would be in September vs. December*, for example.
|
||||
|
||||
- **Logistic regression** helped you discover "binary categories": at this price point, *is this pumpkin orange or not-orange*?
|
||||
|
||||
Classification uses various algorithms to determine other ways of determining a data point's label or class. Let's work with this cuisine data to see whether, by observing a group of ingredients, we can determine its cuisine of origin.
|
||||
|
||||
### [**Pre-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/19/)
|
||||
|
||||
### **Introduction**
|
||||
|
||||
Classification is one of the fundamental activities of the machine learning researcher and data scientist. From basic classification of a binary value ("is this email spam or not?"), to complex image classification and segmentation using computer vision, it's always useful to be able to sort data into classes and ask questions of it.
|
||||
|
||||
To state the process in a more scientific way, your classification method creates a predictive model that enables you to map the relationship between input variables to output variables.
|
||||
|
||||
{width="500"}
|
||||
|
||||
Before starting the process of cleaning our data, visualizing it, and prepping it for our ML tasks, let's learn a bit about the various ways machine learning can be leveraged to classify data.
|
||||
|
||||
Derived from [statistics](https://wikipedia.org/wiki/Statistical_classification), classification using classic machine learning uses features, such as `smoker`, `weight`, and `age` to determine *likelihood of developing X disease*. As a supervised learning technique similar to the regression exercises you performed earlier, your data is labeled and the ML algorithms use those labels to classify and predict classes (or 'features') of a dataset and assign them to a group or outcome.
|
||||
|
||||
✅ Take a moment to imagine a dataset about cuisines. What would a multiclass model be able to answer? What would a binary model be able to answer? What if you wanted to determine whether a given cuisine was likely to use fenugreek? What if you wanted to see if, given a present of a grocery bag full of star anise, artichokes, cauliflower, and horseradish, you could create a typical Indian dish?
|
||||
|
||||
### **Hello 'classifier'**
|
||||
|
||||
The question we want to ask of this cuisine dataset is actually a **multiclass question**, as we have several potential national cuisines to work with. Given a batch of ingredients, which of these many classes will the data fit?
|
||||
|
||||
Tidymodels offers several different algorithms to use to classify data, depending on the kind of problem you want to solve. In the next two lessons, you'll learn about several of these algorithms.
|
||||
|
||||
#### **Prerequisite**
|
||||
|
||||
For this lesson, we'll require the following packages to clean, prep and visualize our data:
|
||||
|
||||
- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!
|
||||
|
||||
- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.
|
||||
|
||||
- `DataExplorer`: The [DataExplorer package](https://cran.r-project.org/web/packages/DataExplorer/vignettes/dataexplorer-intro.html) is meant to simplify and automate EDA process and report generation.
|
||||
|
||||
- `themis`: The [themis package](https://themis.tidymodels.org/) provides Extra Recipes Steps for Dealing with Unbalanced Data.
|
||||
|
||||
You can have them installed as:
|
||||
|
||||
`install.packages(c("tidyverse", "tidymodels", "DataExplorer", "here"))`
|
||||
|
||||
Alternately, the script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.
|
||||
|
||||
```{r, message=F, warning=F}
|
||||
suppressWarnings(if (!require("pacman"))install.packages("pacman"))
|
||||
|
||||
pacman::p_load(tidyverse, tidymodels, DataExplorer, themis, here)
|
||||
```
|
||||
|
||||
We'll later load these awesome packages and make them available in our current R session. (This is for mere illustration, `pacman::p_load()` already did that for you)
|
||||
|
||||
## Exercise - clean and balance your data
|
||||
|
||||
The first task at hand, before starting this project, is to clean and **balance** your data to get better results
|
||||
|
||||
Let's meet the data!🕵️
|
||||
|
||||
```{r import_data}
|
||||
# Import data
|
||||
df <- read_csv(file = "https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/4-Classification/data/cuisines.csv")
|
||||
|
||||
# View the first 5 rows
|
||||
df %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
```
|
||||
|
||||
Interesting! From the looks of it, the first column is a kind of `id` column. Let's get a little more information about the data.
|
||||
|
||||
```{r info}
|
||||
# Basic information about the data
|
||||
df %>%
|
||||
introduce()
|
||||
|
||||
# Visualize basic information above
|
||||
df %>%
|
||||
plot_intro(ggtheme = theme_light())
|
||||
```
|
||||
|
||||
From the output, we can immediately see that we have `2448` rows and `385` columns and `0` missing values. We also have 1 discrete column, *cuisine*.
|
||||
|
||||
## Exercise - learning about cuisines
|
||||
|
||||
1. Now the work starts to become more interesting. Let's discover the distribution of data, per cuisine.
|
||||
|
||||
```{r filter_cuisine}
|
||||
# Count observations per cuisine
|
||||
df %>%
|
||||
count(cuisine) %>%
|
||||
arrange(n)
|
||||
|
||||
# Plot the distribution
|
||||
theme_set(theme_light())
|
||||
df %>%
|
||||
count(cuisine) %>%
|
||||
ggplot(mapping = aes(x = n, y = reorder(cuisine, -n))) +
|
||||
geom_col(fill = "midnightblue", alpha = 0.7) +
|
||||
ylab("cuisine")
|
||||
```
|
||||
|
||||
There are a finite number of cuisines, but the distribution of data is uneven. You can fix that! Before doing so, explore a little more.
|
||||
|
||||
2. Next, let's assign each cuisine into it's individual tibble and find out how much data is available (rows, columns) per cuisine.
|
||||
|
||||
> A tibble, or tbl_df, is a modern reimagining of the data.frame, keeping what time has proven to be effective, and throwing out what is not.
|
||||
|
||||
|
||||
|
||||
```{r cuisine_df}
|
||||
# Create individual tibbles for the cuisines
|
||||
thai_df <- df %>%
|
||||
filter(cuisine == "thai")
|
||||
japanese_df <- df %>%
|
||||
filter(cuisine == "japanese")
|
||||
chinese_df <- df %>%
|
||||
filter(cuisine == "chinese")
|
||||
indian_df <- df %>%
|
||||
filter(cuisine == "indian")
|
||||
korean_df <- df %>%
|
||||
filter(cuisine == "korean")
|
||||
|
||||
|
||||
# Find out how much data is available per cuisine
|
||||
cat(" thai df:", dim(thai_df), "\n",
|
||||
"japanese df:", dim(japanese_df), "\n",
|
||||
"chinese_df:", dim(chinese_df), "\n",
|
||||
"indian_df:", dim(indian_df), "\n",
|
||||
"korean_df:", dim(korean_df))
|
||||
```
|
||||
|
||||
Perfect!😋
|
||||
|
||||
## **Exercise - Discovering top ingredients by cuisine using dplyr**
|
||||
|
||||
Now you can dig deeper into the data and learn what are the typical ingredients per cuisine. You should clean out recurrent data that creates confusion between cuisines, so let's learn about this problem.
|
||||
|
||||
1. Create a function `create_ingredient()` in R that returns an ingredient dataframe. This function will start by dropping an unhelpful column and sort through ingredients by their count.
|
||||
|
||||
The basic structure of a function in R is:
|
||||
|
||||
`myFunction <- function(arglist){`
|
||||
|
||||
**`...`**
|
||||
|
||||
**`return`**`(value)`
|
||||
|
||||
`}`
|
||||
|
||||
A tidy introduction to R functions can be found [here](https://skirmer.github.io/presentations/functions_with_r.html#1).
|
||||
|
||||
Let's get right into it! We'll make use of [dplyr verbs](https://dplyr.tidyverse.org/) which we have been learning in our previous lessons. As a recap:
|
||||
|
||||
- `dplyr::select()`: help you pick which **columns** to keep or exclude.
|
||||
|
||||
- `dplyr::pivot_longer()`: helps you to "lengthen" data, increasing the number of rows and decreasing the number of columns.
|
||||
|
||||
- `dplyr::group_by()` and `dplyr::summarise()`: helps you to find find summary statistics for different groups, and put them in a nice table.
|
||||
|
||||
- `dplyr::filter()`: creates a subset of the data only containing rows that satisfy your conditions.
|
||||
|
||||
- `dplyr::mutate()`: helps you to create or modify columns.
|
||||
|
||||
Check out this [*art*-filled learnr tutorial](https://allisonhorst.shinyapps.io/dplyr-learnr/#section-welcome) by Allison Horst, that introduces some useful data wrangling functions in dplyr *(part of the Tidyverse)*
|
||||
|
||||
```{r create_ingredient}
|
||||
# Creates a functions that returns the top ingredients by class
|
||||
|
||||
create_ingredient <- function(df){
|
||||
|
||||
# Drop the id column which is the first colum
|
||||
ingredient_df = df %>% select(-1) %>%
|
||||
# Transpose data to a long format
|
||||
pivot_longer(!cuisine, names_to = "ingredients", values_to = "count") %>%
|
||||
# Find the top most ingredients for a particular cuisine
|
||||
group_by(ingredients) %>%
|
||||
summarise(n_instances = sum(count)) %>%
|
||||
filter(n_instances != 0) %>%
|
||||
# Arrange by descending order
|
||||
arrange(desc(n_instances)) %>%
|
||||
mutate(ingredients = factor(ingredients) %>% fct_inorder())
|
||||
|
||||
|
||||
return(ingredient_df)
|
||||
} # End of function
|
||||
|
||||
```
|
||||
|
||||
2. Now we can use the function to get an idea of top ten most popular ingredient by cuisine. Let's take it out for a spin with `thai_df`
|
||||
|
||||
```{r thai_ingredient_df}
|
||||
# Call create_ingredient and display popular ingredients
|
||||
thai_ingredient_df <- create_ingredient(df = thai_df)
|
||||
|
||||
thai_ingredient_df %>%
|
||||
slice_head(n = 10)
|
||||
|
||||
```
|
||||
|
||||
In the previous section, we used `geom_col()`, let's see how you can use `geom_bar` too, to create bar charts. Use `?geom_bar` for further reading.
|
||||
|
||||
```{r thai_chart}
|
||||
# Make a bar chart for popular thai cuisines
|
||||
thai_ingredient_df %>%
|
||||
slice_head(n = 10) %>%
|
||||
ggplot(aes(x = n_instances, y = ingredients)) +
|
||||
geom_bar(stat = "identity", width = 0.5, fill = "steelblue") +
|
||||
xlab("") + ylab("")
|
||||
|
||||
```
|
||||
|
||||
3. Let's do the same for the Japanese data
|
||||
|
||||
```{r japanese_ingredient_df}
|
||||
# Get popular ingredients for Japanese cuisines and make bar chart
|
||||
create_ingredient(df = japanese_df) %>%
|
||||
slice_head(n = 10) %>%
|
||||
ggplot(aes(x = n_instances, y = ingredients)) +
|
||||
geom_bar(stat = "identity", width = 0.5, fill = "darkorange", alpha = 0.8) +
|
||||
xlab("") + ylab("")
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
4. What about the Chinese cuisines?
|
||||
|
||||
```{r chinese_ingredient_df}
|
||||
# Get popular ingredients for Chinese cuisines and make bar chart
|
||||
create_ingredient(df = chinese_df) %>%
|
||||
slice_head(n = 10) %>%
|
||||
ggplot(aes(x = n_instances, y = ingredients)) +
|
||||
geom_bar(stat = "identity", width = 0.5, fill = "cyan4", alpha = 0.8) +
|
||||
xlab("") + ylab("")
|
||||
|
||||
|
||||
```
|
||||
|
||||
5. Let's take a look at the Indian cuisines 🌶️.
|
||||
|
||||
```{r indian_ingredient_df }
|
||||
# Get popular ingredients for Indian cuisines and make bar chart
|
||||
create_ingredient(df = indian_df) %>%
|
||||
slice_head(n = 10) %>%
|
||||
ggplot(aes(x = n_instances, y = ingredients)) +
|
||||
geom_bar(stat = "identity", width = 0.5, fill = "#041E42FF", alpha = 0.8) +
|
||||
xlab("") + ylab("")
|
||||
```
|
||||
|
||||
6. Finally, plot the Korean ingredients.
|
||||
|
||||
```{r korean_ingredient_df }
|
||||
# Get popular ingredients for Korean cuisines and make bar chart
|
||||
create_ingredient(df = korean_df) %>%
|
||||
slice_head(n = 10) %>%
|
||||
ggplot(aes(x = n_instances, y = ingredients)) +
|
||||
geom_bar(stat = "identity", width = 0.5, fill = "#852419FF", alpha = 0.8) +
|
||||
xlab("") + ylab("")
|
||||
```
|
||||
|
||||
7. From the data visualizations, we can now drop the most common ingredients that create confusion between distinct cuisines, using `dplyr::select()`.
|
||||
|
||||
Everyone loves rice, garlic and ginger!
|
||||
|
||||
```{r df_select}
|
||||
# Drop rice, garlic and ginger from our original data set
|
||||
df_select <- df %>%
|
||||
select(-c(1, rice, garlic, ginger))
|
||||
|
||||
# Display new data set
|
||||
df_select %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
```
|
||||
|
||||
## Preprocessing data using recipes 👩🍳👨🍳 - Dealing with imbalanced data ⚖️
|
||||
|
||||
|
||||
|
||||
Given that this lesson is about cuisines, we have to put `recipes` into context .
|
||||
|
||||
Tidymodels provides yet another neat package: `recipes`- a package for preprocessing data.
|
||||
|
||||
Now we are on the same page 😅.
|
||||
|
||||
Let's take a look at the distribution of our cuisines again.
|
||||
|
||||
```{r df_select_n}
|
||||
# Distribution of cuisines
|
||||
old_label_count <- df_select %>%
|
||||
count(cuisine) %>%
|
||||
arrange(desc(n))
|
||||
|
||||
old_label_count
|
||||
```
|
||||
|
||||
As you can see, there is quite an unequal distribution in the number of cuisines. Korean cuisines are almost 3 times Thai cuisines. Imbalanced data often has negative effects on the model performance. Think about a binary classification. If most of your data is one class, a ML model is going to predict that class more frequently, just because there is more data for it. Balancing the data takes any skewed data and helps remove this imbalance. Many models perform best when the number of observations is equal and, thus, tend to struggle with unbalanced data.
|
||||
|
||||
There are majorly two ways of dealing with imbalanced data sets:
|
||||
|
||||
- adding observations to the minority class: `Over-sampling` e.g using a SMOTE algorithm
|
||||
|
||||
- removing observations from majority class: `Under-sampling`
|
||||
|
||||
Let's now demonstrate how to deal with imbalanced data sets using a `recipe`. A recipe can be thought of as a blueprint that describes what steps should be applied to a data set in order to get it ready for data analysis.
|
||||
|
||||
```{r recipe}
|
||||
# Load themis package for dealing with imbalanced data
|
||||
library(themis)
|
||||
|
||||
# Create a recipe for preprocessing data
|
||||
cuisines_recipe <- recipe(cuisine ~ ., data = df_select) %>%
|
||||
step_smote(cuisine)
|
||||
|
||||
cuisines_recipe
|
||||
```
|
||||
|
||||
Let's break down our preprocessing steps.
|
||||
|
||||
- The call to `recipe()` with a formula tells the recipe the *roles* of the variables using `df_select` data as the reference. For instance the `cuisine` column has been assigned an `outcome` role while the rest of the columns have been assigned a `predictor` role.
|
||||
|
||||
- [`step_smote(cuisine)`](https://themis.tidymodels.org/reference/step_smote.html) creates a *specification* of a recipe step that synthetically generates new examples of the minority class using nearest neighbors of these cases.
|
||||
|
||||
Now, if we wanted to see the preprocessed data, we'd have to [**`prep()`**](https://recipes.tidymodels.org/reference/prep.html) and [**`bake()`**](https://recipes.tidymodels.org/reference/bake.html) our recipe.
|
||||
|
||||
`prep()`: estimates the required parameters from a training set that can be later applied to other data sets.
|
||||
|
||||
`bake()`: takes a prepped recipe and applies the operations to any data set.
|
||||
|
||||
```{r prep_bake}
|
||||
# Prep and bake the recipe
|
||||
preprocessed_df <- cuisines_recipe %>%
|
||||
prep() %>%
|
||||
bake(new_data = NULL) %>%
|
||||
relocate(cuisine)
|
||||
|
||||
# Display data
|
||||
preprocessed_df %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
# Quick summary stats
|
||||
preprocessed_df %>%
|
||||
introduce()
|
||||
|
||||
```
|
||||
|
||||
Let's now check the distribution of our cuisines and compare them with the imbalanced data.
|
||||
|
||||
```{r prep_cuisines}
|
||||
# Distribution of cuisines
|
||||
new_label_count <- preprocessed_df %>%
|
||||
count(cuisine) %>%
|
||||
arrange(desc(n))
|
||||
|
||||
list(new_label_count = new_label_count,
|
||||
old_label_count = old_label_count)
|
||||
|
||||
```
|
||||
|
||||
Yum! The data is nice and clean, balanced, and very delicious 😋!
|
||||
|
||||
> Normally, a recipe is usually used as a preprocessor for modelling where it defines what steps should be applied to a data set in order to get it ready for modelling. In that case, a `workflow()` is typically used (as we have already seen in our previous lessons) instead of manually estimating a recipe
|
||||
>
|
||||
> As such, you don't typically need to **`prep()`** and **`bake()`** recipes when you use tidymodels, but they are helpful functions to have in your toolkit for confirming that recipes are doing what you expect like in our case.
|
||||
>
|
||||
> When you **`bake()`** a prepped recipe with **`new_data = NULL`**, you get the data that you provided when defining the recipe back, but having undergone the preprocessing steps.
|
||||
|
||||
Let's now save a copy of this data for use in future lessons:
|
||||
|
||||
```{r save_preproc_data}
|
||||
# Save preprocessed data
|
||||
write_csv(preprocessed_df, "../../../data/cleaned_cuisines_R.csv")
|
||||
|
||||
```
|
||||
|
||||
This fresh CSV can now be found in the root data folder.
|
||||
|
||||
**🚀Challenge**
|
||||
|
||||
This curriculum contains several interesting datasets. Dig through the `data` folders and see if any contain datasets that would be appropriate for binary or multi-class classification? What questions would you ask of this dataset?
|
||||
|
||||
## [**Post-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/20/)
|
||||
|
||||
## **Review & Self Study**
|
||||
|
||||
- Check out [package themis](https://github.com/tidymodels/themis). What other techniques could we use to deal with imbalanced data?
|
||||
|
||||
- Tidy models [reference website](https://www.tidymodels.org/start/).
|
||||
|
||||
- H. Wickham and G. Grolemund, [*R for Data Science: Visualize, Model, Transform, Tidy, and Import Data*](https://r4ds.had.co.nz/).
|
||||
|
||||
#### THANK YOU TO:
|
||||
|
||||
[`Allison Horst`](https://twitter.com/allison_horst/) for creating the amazing illustrations that make R more welcoming and engaging. Find more illustrations at her [gallery](https://www.google.com/url?q=https://github.com/allisonhorst/stats-illustrations&sa=D&source=editors&ust=1626380772530000&usg=AOvVaw3zcfyCizFQZpkSLzxiiQEM).
|
||||
|
||||
[Cassie Breviu](https://www.twitter.com/cassieview) and [Jen Looper](https://www.twitter.com/jenlooper) for creating the original Python version of this module ♥️
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user