chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,21 @@
|
||||
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.203.0/containers/python-3/.devcontainer/base.Dockerfile
|
||||
|
||||
# [Choice] Python version (use -bullseye variants on local arm64/Apple Silicon): 3, 3.10, 3.9, 3.8, 3.7, 3.6, 3-bullseye, 3.10-bullseye, 3.9-bullseye, 3.8-bullseye, 3.7-bullseye, 3.6-bullseye, 3-buster, 3.10-buster, 3.9-buster, 3.8-buster, 3.7-buster, 3.6-buster
|
||||
ARG VARIANT="3.10-bullseye"
|
||||
FROM mcr.microsoft.com/vscode/devcontainers/python:0-${VARIANT}
|
||||
|
||||
# [Choice] Node.js version: none, lts/*, 16, 14, 12, 10
|
||||
ARG NODE_VERSION="none"
|
||||
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi
|
||||
|
||||
# [Optional] If your pip requirements rarely change, uncomment this section to add them to the image.
|
||||
# COPY requirements.txt /tmp/pip-tmp/
|
||||
# RUN pip3 --disable-pip-version-check --no-cache-dir install -r /tmp/pip-tmp/requirements.txt \
|
||||
# && rm -rf /tmp/pip-tmp
|
||||
|
||||
# [Optional] Uncomment this section to install additional OS packages.
|
||||
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
|
||||
# && apt-get -y install --no-install-recommends <your-package-list-here>
|
||||
|
||||
# [Optional] Uncomment this line to install global node packages.
|
||||
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1
|
||||
@@ -0,0 +1,50 @@
|
||||
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
|
||||
// https://github.com/microsoft/vscode-dev-containers/tree/v0.203.0/containers/python-3
|
||||
{
|
||||
"name": "Python 3",
|
||||
"runArgs": ["--init"],
|
||||
"build": {
|
||||
"dockerfile": "Dockerfile",
|
||||
"context": "..",
|
||||
"args": {
|
||||
// Update 'VARIANT' to pick a Python version: 3, 3.10, 3.9, 3.8, 3.7, 3.6
|
||||
// Append -bullseye or -buster to pin to an OS version.
|
||||
// Use -bullseye variants on local on arm64/Apple Silicon.
|
||||
"VARIANT": "3",
|
||||
// Options
|
||||
"NODE_VERSION": "none"
|
||||
}
|
||||
},
|
||||
|
||||
// Set *default* container specific settings.json values on container create.
|
||||
"settings": {
|
||||
"python.pythonPath": "/usr/local/bin/python",
|
||||
"python.languageServer": "Pylance",
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.pylintEnabled": true,
|
||||
"python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8",
|
||||
"python.formatting.blackPath": "/usr/local/py-utils/bin/black",
|
||||
"python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf",
|
||||
"python.linting.banditPath": "/usr/local/py-utils/bin/bandit",
|
||||
"python.linting.flake8Path": "/usr/local/py-utils/bin/flake8",
|
||||
"python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
|
||||
"python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle",
|
||||
"python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle",
|
||||
"python.linting.pylintPath": "/usr/local/py-utils/bin/pylint"
|
||||
},
|
||||
|
||||
// Add the IDs of extensions you want installed when the container is created.
|
||||
"extensions": [
|
||||
"ms-python.python",
|
||||
"ms-python.vscode-pylance"
|
||||
],
|
||||
|
||||
// Use 'forwardPorts' to make a list of ports inside the container available locally.
|
||||
// "forwardPorts": [],
|
||||
|
||||
// Use 'postCreateCommand' to run commands after the container is created.
|
||||
// "postCreateCommand": "pip3 install --user -r requirements.txt",
|
||||
|
||||
// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
|
||||
"remoteUser": "vscode"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
---
|
||||
name: Bug report
|
||||
about: Create a report to help us improve
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Describe the bug**
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
**To Reproduce**
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '....'
|
||||
3. Scroll down to '....'
|
||||
4. See error
|
||||
|
||||
**Expected behavior**
|
||||
A clear and concise description of what you expected to happen.
|
||||
|
||||
**Screenshots**
|
||||
If applicable, add screenshots to help explain your problem.
|
||||
|
||||
**Desktop (please complete the following information):**
|
||||
- OS: [e.g. iOS]
|
||||
- Browser [e.g. chrome, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Smartphone (please complete the following information):**
|
||||
- Device: [e.g. iPhone6]
|
||||
- OS: [e.g. iOS8.1]
|
||||
- Browser [e.g. stock browser, safari]
|
||||
- Version [e.g. 22]
|
||||
|
||||
**Additional context**
|
||||
Add any other context about the problem here.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
name: Feature request
|
||||
about: Suggest an idea for this project
|
||||
title: ''
|
||||
labels: ''
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
**Is your feature request related to a problem? Please describe.**
|
||||
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
|
||||
|
||||
**Describe the solution you'd like**
|
||||
A clear and concise description of what you want to happen.
|
||||
|
||||
**Describe alternatives you've considered**
|
||||
A clear and concise description of any alternative solutions or features you've considered.
|
||||
|
||||
**Additional context**
|
||||
Add any other context or screenshots about the feature request here.
|
||||
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: Translations Checklist
|
||||
about: These are all the files that need to be translated
|
||||
title: "[TRANSLATIONS]"
|
||||
labels: translations
|
||||
assignees: ''
|
||||
|
||||
---
|
||||
|
||||
- [ ] Base README.md
|
||||
- [ ] Quizzes
|
||||
- [ ] Introduction base README
|
||||
- [ ] Intro to ML README
|
||||
- [ ] Intro to ML assignment
|
||||
- [ ] History of ML README
|
||||
- [ ] History of ML assignment
|
||||
- [ ] Fairness README
|
||||
- [ ] Fairness assignment
|
||||
- [ ] Techniques of ML README
|
||||
- [ ] Techniques of ML assignment
|
||||
- [ ] Regression base README
|
||||
- [ ] Tools README
|
||||
- [ ] Tools assignment
|
||||
- [ ] Data README
|
||||
- [ ] Data assignment
|
||||
- [ ] Linear README
|
||||
- [ ] Linear assignment
|
||||
- [ ] Logistic README
|
||||
- [ ] Logistic assignment
|
||||
- [ ] Web app base README
|
||||
- [ ] Web app README
|
||||
- [ ] Web app assignment
|
||||
- [ ] Classification base README
|
||||
- [ ] Intro README
|
||||
- [ ] Intro assignment
|
||||
- [ ] Classifiers 1 README
|
||||
- [ ] Classifiers 1 assignment
|
||||
- [ ] Classifiers 2 README
|
||||
- [ ] Classifiers 2 assignment
|
||||
- [ ] Applied README
|
||||
- [ ] Applied assignment
|
||||
- [ ] Clustering base README
|
||||
- [ ] Visualize README
|
||||
- [ ] Visualize assignment
|
||||
- [ ] K-means README
|
||||
- [ ] K-means assignment
|
||||
- [ ] NLP base README
|
||||
- [ ] Intro README
|
||||
- [ ] Intro assignment
|
||||
- [ ] Tasks README
|
||||
- [ ] Tasks assignment
|
||||
- [ ] Translation README
|
||||
- [ ] Translation assignment
|
||||
- [ ] Reviews 1 README
|
||||
- [ ] Reviews 1 assignment
|
||||
- [ ] Reviews 2 README
|
||||
- [ ] Reviews 2 assignment
|
||||
- [ ] Time Series base README
|
||||
- [ ] Intro README
|
||||
- [ ] Intro assignment
|
||||
- [ ] ARIMA README
|
||||
- [ ] ARIMA assignment
|
||||
- [ ] Reinforcement base README
|
||||
- [ ] QLearning README
|
||||
- [ ] QLearning assignment
|
||||
- [ ] gym README
|
||||
- [ ] gym assignment
|
||||
- [ ] Real World base README
|
||||
- [ ] Real World README
|
||||
- [ ] Real World assignment
|
||||
@@ -0,0 +1,37 @@
|
||||
## Track translation progress by opening a draft PR using this template and checking off the translations completed
|
||||
|
||||
Each lesson includes a translation of the README.md and the Assignment.md file, if available. Only mark the lesson complete if both those files are translated per lesson, please.
|
||||
|
||||
- [ ] 1
|
||||
- [ ] 1-1
|
||||
- [ ] 1-2
|
||||
- [ ] 1-3
|
||||
- [ ] 2
|
||||
- [ ] 2-1
|
||||
- [ ] 2-2
|
||||
- [ ] 2-3
|
||||
- [ ] 2-4
|
||||
- [ ] 3
|
||||
- [ ] 3-1
|
||||
- [ ] 3-2
|
||||
- [ ] 3-3
|
||||
- [ ] 4
|
||||
- [ ] 4-1
|
||||
- [ ] 5
|
||||
- [ ] 5-1
|
||||
- [ ] 5-2
|
||||
- [ ] 5-3
|
||||
- [ ] 6
|
||||
- [ ] 6-1
|
||||
- [ ] 6-2
|
||||
- [ ] 6-3
|
||||
- [ ] 6-4
|
||||
- [ ] 6-5
|
||||
- [ ] 6-6
|
||||
- [ ] 7
|
||||
- [ ] 7-1
|
||||
- [ ] 7-2
|
||||
- [ ] 7-3
|
||||
- [ ] 7-4
|
||||
|
||||
- [ ] Quiz (add a file in the quiz-app with all localizations)
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: 'A chat mode to help you as you learn.'
|
||||
tools: []
|
||||
---
|
||||
You are my Adaptive Study Coach inside this codebase.
|
||||
|
||||
Goals
|
||||
- Help me learn {TOPIC} by doing, not just reading.
|
||||
- Use the repo/workspace context when giving examples or tasks.
|
||||
- Keep replies concise (≤120 words) unless I say “expand”.
|
||||
|
||||
Protocol
|
||||
1) Kickoff: ask my level (Beginner/Intermediate/Advanced), timebox (e.g., 25m), and one concrete goal.
|
||||
2) Loop each cycle:
|
||||
a) 60-sec concept recap tied to the repo.
|
||||
b) 3 progressive questions (Socratic). Offer HINT 1/2/3 on request.
|
||||
c) 1 micro-task (≤10 lines of code). Wait for my attempt before revealing.
|
||||
d) Feedback: what’s right, what to improve, 1 tiny next step.
|
||||
3) Use file/line references and unified diffs when suggesting edits.
|
||||
4) Never dump full solutions unless I say “reveal”.
|
||||
5) After each cycle, give: (i) 3 bullet takeaways, (ii) 2 next micro-tasks.
|
||||
6) End of session: output 5 spaced-repetition flashcards (Anki cloze format) + a 5-item checklist for tomorrow.
|
||||
|
||||
Constraints & Style
|
||||
- One question at a time.
|
||||
- Prefer examples drawn from this repo or minimal snippets.
|
||||
- If I seem stuck, switch to “guided mode”: smaller hints, then partial skeletons, then solution.
|
||||
- Be encouraging but exact; correct misconceptions immediately.
|
||||
|
||||
Commands I’ll use
|
||||
- “hint”, “next”, “reveal”, “harder”, “slower”, “quiz me”, “make flashcards”, “give me a diff”.Cl
|
||||
@@ -0,0 +1,66 @@
|
||||
# This workflow uses actions that are not certified by GitHub.
|
||||
# They are provided by a third-party and are governed by
|
||||
# separate terms of service, privacy policy, and support
|
||||
# documentation.
|
||||
|
||||
# This workflow lets you generate SLSA provenance file for your project.
|
||||
# The generation satisfies level 3 for the provenance requirements - see https://slsa.dev/spec/v0.1/requirements
|
||||
# The project is an initiative of the OpenSSF (openssf.org) and is developed at
|
||||
# https://github.com/slsa-framework/slsa-github-generator.
|
||||
# The provenance file can be verified using https://github.com/slsa-framework/slsa-verifier.
|
||||
# For more information about SLSA and how it improves the supply-chain, visit slsa.dev.
|
||||
|
||||
name: SLSA generic generator
|
||||
on:
|
||||
workflow_dispatch:
|
||||
release:
|
||||
types: [created]
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
digests: ${{ steps.hash.outputs.digests }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
# ========================================================
|
||||
#
|
||||
# Step 1: Build your artifacts.
|
||||
#
|
||||
# ========================================================
|
||||
- name: Build artifacts
|
||||
run: |
|
||||
# These are some amazing artifacts.
|
||||
echo "artifact1" > artifact1
|
||||
echo "artifact2" > artifact2
|
||||
|
||||
# ========================================================
|
||||
#
|
||||
# Step 2: Add a step to generate the provenance subjects
|
||||
# as shown below. Update the sha256 sum arguments
|
||||
# to include all binaries that you generate
|
||||
# provenance for.
|
||||
#
|
||||
# ========================================================
|
||||
- name: Generate subject for provenance
|
||||
id: hash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# List the artifacts the provenance will refer to.
|
||||
files=$(ls artifact*)
|
||||
# Generate the subjects (base64 encoded).
|
||||
echo "hashes=$(sha256sum $files | base64 -w0)" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
provenance:
|
||||
needs: [build]
|
||||
permissions:
|
||||
actions: read # To read the workflow path.
|
||||
id-token: write # To sign the provenance.
|
||||
contents: write # To add assets to a release.
|
||||
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.4.0
|
||||
with:
|
||||
base64-subjects: "${{ needs.build.outputs.digests }}"
|
||||
upload-assets: true # Optional: Upload to a new release
|
||||
@@ -0,0 +1,13 @@
|
||||
name: Lock closed issue
|
||||
|
||||
on:
|
||||
issues:
|
||||
types: [closed]
|
||||
|
||||
jobs:
|
||||
lock:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: OSDKDev/lock-issues@v1.1
|
||||
with:
|
||||
repo-token: "${{ secrets.GITHUB_TOKEN }}"
|
||||
@@ -0,0 +1,27 @@
|
||||
# This workflow warns and then closes issues and PRs that have had no activity for a specified amount of time.
|
||||
#
|
||||
# You can adjust the behavior by modifying this file.
|
||||
# For more information, see:
|
||||
# https://github.com/actions/stale
|
||||
name: Mark stale issues and pull requests
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '35 8 * * *'
|
||||
|
||||
jobs:
|
||||
stale:
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- uses: actions/stale@v10
|
||||
with:
|
||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
stale-issue-message: 'This issue has not seen any action for a while! Closing for now, but it can be reopened at a later date.'
|
||||
stale-pr-message: 'This PR has not seen any action for a while! Closing for now, but it can be reopened at a later date.'
|
||||
stale-issue-label: 'no-issue-activity'
|
||||
stale-pr-label: 'no-pr-activity'
|
||||
+366
@@ -0,0 +1,366 @@
|
||||
## Ignore Visual Studio temporary files, build results, and
|
||||
## files generated by popular Visual Studio add-ons.
|
||||
##
|
||||
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
|
||||
|
||||
dist
|
||||
|
||||
# User-specific files
|
||||
*.rsuser
|
||||
*.suo
|
||||
*.user
|
||||
*.userosscache
|
||||
*.sln.docstates
|
||||
|
||||
# User-specific files (MonoDevelop/Xamarin Studio)
|
||||
*.userprefs
|
||||
|
||||
# Mono auto-generated files
|
||||
mono_crash.*
|
||||
|
||||
# Build results
|
||||
[Dd]ebug/
|
||||
[Dd]ebugPublic/
|
||||
[Rr]elease/
|
||||
[Rr]eleases/
|
||||
x64/
|
||||
x86/
|
||||
[Aa][Rr][Mm]/
|
||||
[Aa][Rr][Mm]64/
|
||||
bld/
|
||||
[Bb]in/
|
||||
[Oo]bj/
|
||||
[Ll]og/
|
||||
[Ll]ogs/
|
||||
|
||||
# Visual Studio 2015/2017 cache/options directory
|
||||
.vs/
|
||||
# Visual Studio Code cache/options directory
|
||||
.vscode/
|
||||
# Uncomment if you have tasks that create the project's static files in wwwroot
|
||||
#wwwroot/
|
||||
|
||||
# Visual Studio 2017 auto-generated files
|
||||
Generated\ Files/
|
||||
|
||||
# MSTest test Results
|
||||
[Tt]est[Rr]esult*/
|
||||
[Bb]uild[Ll]og.*
|
||||
|
||||
# NUnit
|
||||
*.VisualState.xml
|
||||
TestResult.xml
|
||||
nunit-*.xml
|
||||
|
||||
# Build Results of an ATL Project
|
||||
[Dd]ebugPS/
|
||||
[Rr]eleasePS/
|
||||
dlldata.c
|
||||
|
||||
# Benchmark Results
|
||||
BenchmarkDotNet.Artifacts/
|
||||
|
||||
# .NET Core
|
||||
project.lock.json
|
||||
project.fragment.lock.json
|
||||
artifacts/
|
||||
|
||||
# StyleCop
|
||||
StyleCopReport.xml
|
||||
|
||||
# Files built by Visual Studio
|
||||
*_i.c
|
||||
*_p.c
|
||||
*_h.h
|
||||
*.ilk
|
||||
*.meta
|
||||
*.obj
|
||||
*.iobj
|
||||
*.pch
|
||||
*.pdb
|
||||
*.ipdb
|
||||
*.pgc
|
||||
*.pgd
|
||||
*.rsp
|
||||
*.sbr
|
||||
*.tlb
|
||||
*.tli
|
||||
*.tlh
|
||||
*.tmp
|
||||
*.tmp_proj
|
||||
*_wpftmp.csproj
|
||||
*.log
|
||||
*.vspscc
|
||||
*.vssscc
|
||||
.builds
|
||||
*.pidb
|
||||
*.svclog
|
||||
*.scc
|
||||
|
||||
# Chutzpah Test files
|
||||
_Chutzpah*
|
||||
|
||||
# Visual C++ cache files
|
||||
ipch/
|
||||
*.aps
|
||||
*.ncb
|
||||
*.opendb
|
||||
*.opensdf
|
||||
*.sdf
|
||||
*.cachefile
|
||||
*.VC.db
|
||||
*.VC.VC.opendb
|
||||
|
||||
# Visual Studio profiler
|
||||
*.psess
|
||||
*.vsp
|
||||
*.vspx
|
||||
*.sap
|
||||
|
||||
# Visual Studio Trace Files
|
||||
*.e2e
|
||||
|
||||
# TFS 2012 Local Workspace
|
||||
$tf/
|
||||
|
||||
# Guidance Automation Toolkit
|
||||
*.gpState
|
||||
|
||||
# ReSharper is a .NET coding add-in
|
||||
_ReSharper*/
|
||||
*.[Rr]e[Ss]harper
|
||||
*.DotSettings.user
|
||||
|
||||
# TeamCity is a build add-in
|
||||
_TeamCity*
|
||||
|
||||
# DotCover is a Code Coverage Tool
|
||||
*.dotCover
|
||||
|
||||
# AxoCover is a Code Coverage Tool
|
||||
.axoCover/*
|
||||
!.axoCover/settings.json
|
||||
|
||||
# Visual Studio code coverage results
|
||||
*.coverage
|
||||
*.coveragexml
|
||||
|
||||
# NCrunch
|
||||
_NCrunch_*
|
||||
.*crunch*.local.xml
|
||||
nCrunchTemp_*
|
||||
|
||||
# MightyMoose
|
||||
*.mm.*
|
||||
AutoTest.Net/
|
||||
|
||||
# Web workbench (sass)
|
||||
.sass-cache/
|
||||
|
||||
# Installshield output folder
|
||||
[Ee]xpress/
|
||||
|
||||
# DocProject is a documentation generator add-in
|
||||
DocProject/buildhelp/
|
||||
DocProject/Help/*.HxT
|
||||
DocProject/Help/*.HxC
|
||||
DocProject/Help/*.hhc
|
||||
DocProject/Help/*.hhk
|
||||
DocProject/Help/*.hhp
|
||||
DocProject/Help/Html2
|
||||
DocProject/Help/html
|
||||
|
||||
# Click-Once directory
|
||||
publish/
|
||||
|
||||
# Publish Web Output
|
||||
*.[Pp]ublish.xml
|
||||
*.azurePubxml
|
||||
# Note: Comment the next line if you want to checkin your web deploy settings,
|
||||
# but database connection strings (with potential passwords) will be unencrypted
|
||||
*.pubxml
|
||||
*.publishproj
|
||||
|
||||
# Microsoft Azure Web App publish settings. Comment the next line if you want to
|
||||
# checkin your Azure Web App publish settings, but sensitive information contained
|
||||
# in these scripts will be unencrypted
|
||||
PublishScripts/
|
||||
|
||||
# NuGet Packages
|
||||
*.nupkg
|
||||
# NuGet Symbol Packages
|
||||
*.snupkg
|
||||
# The packages folder can be ignored because of Package Restore
|
||||
**/[Pp]ackages/*
|
||||
# except build/, which is used as an MSBuild target.
|
||||
!**/[Pp]ackages/build/
|
||||
# Uncomment if necessary however generally it will be regenerated when needed
|
||||
#!**/[Pp]ackages/repositories.config
|
||||
# NuGet v3's project.json files produces more ignorable files
|
||||
*.nuget.props
|
||||
*.nuget.targets
|
||||
|
||||
# Microsoft Azure Build Output
|
||||
csx/
|
||||
*.build.csdef
|
||||
|
||||
# Microsoft Azure Emulator
|
||||
ecf/
|
||||
rcf/
|
||||
|
||||
# Windows Store app package directories and files
|
||||
AppPackages/
|
||||
BundleArtifacts/
|
||||
Package.StoreAssociation.xml
|
||||
_pkginfo.txt
|
||||
*.appx
|
||||
*.appxbundle
|
||||
*.appxupload
|
||||
|
||||
# Visual Studio cache files
|
||||
# files ending in .cache can be ignored
|
||||
*.[Cc]ache
|
||||
# but keep track of directories ending in .cache
|
||||
!?*.[Cc]ache/
|
||||
|
||||
# Others
|
||||
ClientBin/
|
||||
~$*
|
||||
*~
|
||||
*.dbmdl
|
||||
*.dbproj.schemaview
|
||||
*.jfm
|
||||
*.pfx
|
||||
*.publishsettings
|
||||
orleans.codegen.cs
|
||||
|
||||
# Including strong name files can present a security risk
|
||||
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
|
||||
#*.snk
|
||||
|
||||
# Since there are multiple workflows, uncomment next line to ignore bower_components
|
||||
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
|
||||
#bower_components/
|
||||
|
||||
# RIA/Silverlight projects
|
||||
Generated_Code/
|
||||
|
||||
# Backup & report files from converting an old project file
|
||||
# to a newer Visual Studio version. Backup files are not needed,
|
||||
# because we have git ;-)
|
||||
_UpgradeReport_Files/
|
||||
Backup*/
|
||||
UpgradeLog*.XML
|
||||
UpgradeLog*.htm
|
||||
ServiceFabricBackup/
|
||||
*.rptproj.bak
|
||||
|
||||
# SQL Server files
|
||||
*.mdf
|
||||
*.ldf
|
||||
*.ndf
|
||||
|
||||
# Business Intelligence projects
|
||||
*.rdl.data
|
||||
*.bim.layout
|
||||
*.bim_*.settings
|
||||
*.rptproj.rsuser
|
||||
*- [Bb]ackup.rdl
|
||||
*- [Bb]ackup ([0-9]).rdl
|
||||
*- [Bb]ackup ([0-9][0-9]).rdl
|
||||
|
||||
# Microsoft Fakes
|
||||
FakesAssemblies/
|
||||
|
||||
# GhostDoc plugin setting file
|
||||
*.GhostDoc.xml
|
||||
|
||||
# Node.js Tools for Visual Studio
|
||||
.ntvs_analysis.dat
|
||||
node_modules/
|
||||
|
||||
# Visual Studio 6 build log
|
||||
*.plg
|
||||
|
||||
# Visual Studio 6 workspace options file
|
||||
*.opt
|
||||
|
||||
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
|
||||
*.vbw
|
||||
|
||||
# Visual Studio LightSwitch build output
|
||||
**/*.HTMLClient/GeneratedArtifacts
|
||||
**/*.DesktopClient/GeneratedArtifacts
|
||||
**/*.DesktopClient/ModelManifest.xml
|
||||
**/*.Server/GeneratedArtifacts
|
||||
**/*.Server/ModelManifest.xml
|
||||
_Pvt_Extensions
|
||||
|
||||
# Paket dependency manager
|
||||
.paket/paket.exe
|
||||
paket-files/
|
||||
|
||||
# FAKE - F# Make
|
||||
.fake/
|
||||
|
||||
# CodeRush personal settings
|
||||
.cr/personal
|
||||
|
||||
# Python Tools for Visual Studio (PTVS)
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Cake - Uncomment if you are using it
|
||||
# tools/**
|
||||
# !tools/packages.config
|
||||
|
||||
# Tabs Studio
|
||||
*.tss
|
||||
|
||||
# Telerik's JustMock configuration file
|
||||
*.jmconfig
|
||||
|
||||
# BizTalk build output
|
||||
*.btp.cs
|
||||
*.btm.cs
|
||||
*.odx.cs
|
||||
*.xsd.cs
|
||||
|
||||
# OpenCover UI analysis results
|
||||
OpenCover/
|
||||
|
||||
# Azure Stream Analytics local run output
|
||||
ASALocalRun/
|
||||
|
||||
# MSBuild Binary and Structured Log
|
||||
*.binlog
|
||||
|
||||
# NVidia Nsight GPU debugger configuration file
|
||||
*.nvuser
|
||||
|
||||
# MFractors (Xamarin productivity tool) working folder
|
||||
.mfractor/
|
||||
|
||||
# Local History for Visual Studio
|
||||
.localhistory/
|
||||
|
||||
# BeatPulse healthcheck temp database
|
||||
healthchecksdb
|
||||
|
||||
# Backup folder for Package Reference Convert tool in Visual Studio 2017
|
||||
MigrationBackup/
|
||||
|
||||
# Ionide (cross platform F# VS Code tools) working folder
|
||||
.ionide/
|
||||
|
||||
# Mac-specific
|
||||
.DS_Store
|
||||
.Rproj.user
|
||||
|
||||
.Rdata
|
||||
.Rhistory
|
||||
ML-For-Beginners.Rproj
|
||||
|
||||
.env
|
||||
.venv
|
||||
venv
|
||||
@@ -0,0 +1,150 @@
|
||||
# Introduction to machine learning
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
---
|
||||
|
||||
[](https://youtu.be/6mSx_KJxcHI "ML for beginners - Introduction to Machine Learning for Beginners")
|
||||
|
||||
> 🎥 Click the image above for a short video working through this lesson.
|
||||
|
||||
Welcome to this course on classical machine learning for beginners! Whether you're completely new to this topic, or an experienced ML practitioner looking to brush up on an area, we're happy to have you join us! We want to create a friendly launching spot for your ML study and would be happy to evaluate, respond to, and incorporate your [feedback](https://github.com/microsoft/ML-For-Beginners/discussions).
|
||||
|
||||
[](https://youtu.be/h0e2HAPTGF4 "Introduction to ML")
|
||||
|
||||
> 🎥 Click the image above for a video: MIT's John Guttag introduces machine learning
|
||||
|
||||
---
|
||||
## Getting started with machine learning
|
||||
|
||||
Before starting with this curriculum, you need to have your computer set up and ready to run notebooks locally.
|
||||
|
||||
- **Configure your machine with these videos**. Use the following links to learn [how to install Python](https://youtu.be/CXZYvNRIAKM) in your system and [setup a text editor](https://youtu.be/EU8eayHWoZg) for development.
|
||||
- **Learn Python**. It's also recommended to have a basic understanding of [Python](https://docs.microsoft.com/learn/paths/python-language/?WT.mc_id=academic-77952-leestott), a programming language useful for data scientists that we use in this course.
|
||||
- **Learn Node.js and JavaScript**. We also use JavaScript a few times in this course when building web apps, so you will need to have [node](https://nodejs.org) and [npm](https://www.npmjs.com/) installed, as well as [Visual Studio Code](https://code.visualstudio.com/) available for both Python and JavaScript development.
|
||||
- **Create a GitHub account**. Since you found us here on [GitHub](https://github.com), you might already have an account, but if not, create one and then fork this curriculum to use on your own. (Feel free to give us a star, too 😊)
|
||||
- **Explore Scikit-learn**. Familiarize yourself with [Scikit-learn](https://scikit-learn.org/stable/user_guide.html), a set of ML libraries that we reference in these lessons.
|
||||
|
||||
---
|
||||
## What is machine learning?
|
||||
|
||||
The term 'machine learning' is one of the most popular and frequently used terms of today. There is a nontrivial possibility that you have heard this term at least once if you have some sort of familiarity with technology, no matter what domain you work in. The mechanics of machine learning, however, are a mystery to most people. For a machine learning beginner, the subject can sometimes feel overwhelming. Therefore, it is important to understand what machine learning actually is, and to learn about it step by step, through practical examples.
|
||||
|
||||
---
|
||||
## The hype curve
|
||||
|
||||
|
||||
|
||||
> Google Trends shows the recent 'hype curve' of the term 'machine learning'
|
||||
|
||||
---
|
||||
## A mysterious universe
|
||||
|
||||
We live in a universe full of fascinating mysteries. Great scientists such as Stephen Hawking, Albert Einstein, and many more have devoted their lives to searching for meaningful information that uncovers the mysteries of the world around us. This is the human condition of learning: a human child learns new things and uncovers the structure of their world year by year as they grow to adulthood.
|
||||
|
||||
---
|
||||
## The child's brain
|
||||
|
||||
A child's brain and senses perceive the facts of their surroundings and gradually learn the hidden patterns of life which help the child to craft logical rules to identify learned patterns. The learning process of the human brain makes humans the most sophisticated living creature of this world. Learning continuously by discovering hidden patterns and then innovating on those patterns enables us to make ourselves better and better throughout our lifetime. This learning capacity and evolving capability is related to a concept called [brain plasticity](https://www.simplypsychology.org/brain-plasticity.html). Superficially, we can draw some motivational similarities between the learning process of the human brain and the concepts of machine learning.
|
||||
|
||||
---
|
||||
## The human brain
|
||||
|
||||
The [human brain](https://www.livescience.com/29365-human-brain.html) perceives things from the real world, processes the perceived information, makes rational decisions, and performs certain actions based on circumstances. This is what we called behaving intelligently. When we program a facsimile of the intelligent behavioral process to a machine, it is called artificial intelligence (AI).
|
||||
|
||||
---
|
||||
## Some terminology
|
||||
|
||||
Although the terms can be confused, machine learning (ML) is an important subset of artificial intelligence. **ML is concerned with using specialized algorithms to uncover meaningful information and find hidden patterns from perceived data to corroborate the rational decision-making process**.
|
||||
|
||||
---
|
||||
## AI, ML, Deep Learning
|
||||
|
||||
|
||||
|
||||
> A diagram showing the relationships between AI, ML, deep learning, and data science. Infographic by [Jen Looper](https://twitter.com/jenlooper) inspired by [this graphic](https://softwareengineering.stackexchange.com/questions/366996/distinction-between-ai-ml-neural-networks-deep-learning-and-data-mining)
|
||||
|
||||
---
|
||||
## Concepts to cover
|
||||
|
||||
In this curriculum, we are going to cover only the core concepts of machine learning that a beginner must know. We cover what we call 'classical machine learning' primarily using Scikit-learn, an excellent library many students use to learn the basics. To understand broader concepts of artificial intelligence or deep learning, a strong fundamental knowledge of machine learning is indispensable, and so we would like to offer it here.
|
||||
|
||||
---
|
||||
## In this course you will learn:
|
||||
|
||||
- core concepts of machine learning
|
||||
- the history of ML
|
||||
- ML and fairness
|
||||
- regression ML techniques
|
||||
- classification ML techniques
|
||||
- clustering ML techniques
|
||||
- natural language processing ML techniques
|
||||
- time series forecasting ML techniques
|
||||
- reinforcement learning
|
||||
- real-world applications for ML
|
||||
|
||||
---
|
||||
## What we will not cover
|
||||
|
||||
- deep learning
|
||||
- neural networks
|
||||
- AI
|
||||
|
||||
To make for a better learning experience, we will avoid the complexities of neural networks, 'deep learning' - many-layered model-building using neural networks - and AI, which we will discuss in a different curriculum. We also will offer a forthcoming data science curriculum to focus on that aspect of this larger field.
|
||||
|
||||
---
|
||||
## Why study machine learning?
|
||||
|
||||
Machine learning, from a systems perspective, is defined as the creation of automated systems that can learn hidden patterns from data to aid in making intelligent decisions.
|
||||
|
||||
This motivation is loosely inspired by how the human brain learns certain things based on the data it perceives from the outside world.
|
||||
|
||||
✅ Think for a minute why a business would want to try to use machine learning strategies vs. creating a hard-coded rules-based engine.
|
||||
|
||||
---
|
||||
## Why data quality matters
|
||||
|
||||
High-quality data improves model performance. Poor or noisy data can lead to inaccurate predictions, even when using advanced machine learning algorithms.
|
||||
|
||||
---
|
||||
## Applications of machine learning
|
||||
|
||||
Applications of machine learning are now almost everywhere, and are as ubiquitous as the data that is flowing around our societies, generated by our smart phones, connected devices, and other systems. Considering the immense potential of state-of-the-art machine learning algorithms, researchers have been exploring their capability to solve multi-dimensional and multi-disciplinary real-life problems with great positive outcomes.
|
||||
|
||||
---
|
||||
## Examples of applied ML
|
||||
|
||||
**You can use machine learning in many ways**:
|
||||
|
||||
- To predict the likelihood of disease from a patient's medical history or reports.
|
||||
- To leverage weather data to predict weather events.
|
||||
- To understand the sentiment of a text.
|
||||
- To detect fake news to stop the spread of propaganda.
|
||||
|
||||
Finance, economics, earth science, space exploration, biomedical engineering, cognitive science, and even fields in the humanities have adapted machine learning to solve the arduous, data-processing heavy problems of their domain.
|
||||
|
||||
---
|
||||
## Conclusion
|
||||
|
||||
Machine learning automates the process of pattern-discovery by finding meaningful insights from real-world or generated data. It has proven itself to be highly valuable in business, health, and financial applications, among others.
|
||||
|
||||
In the near future, understanding the basics of machine learning is going to be a must for people from any domain due to its widespread adoption.
|
||||
|
||||
---
|
||||
# 🚀 Challenge
|
||||
|
||||
Sketch, on paper or using an online app like [Excalidraw](https://excalidraw.com/), your understanding of the differences between AI, ML, deep learning, and data science. Add some ideas of problems that each of these techniques are good at solving.
|
||||
|
||||
# [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
---
|
||||
# Review & Self Study
|
||||
|
||||
To learn more about how you can work with ML algorithms in the cloud, follow this [Learning Path](https://docs.microsoft.com/learn/paths/create-no-code-predictive-models-azure-machine-learning/?WT.mc_id=academic-77952-leestott).
|
||||
|
||||
Take a [Learning Path](https://docs.microsoft.com/learn/modules/introduction-to-machine-learning/?WT.mc_id=academic-77952-leestott) about the basics of ML.
|
||||
|
||||
---
|
||||
# Assignment
|
||||
|
||||
[Get up and running](assignment.md)
|
||||
@@ -0,0 +1,9 @@
|
||||
# Get Up and Running
|
||||
|
||||
## Instructions
|
||||
|
||||
In this non-graded assignment, you should brush up on Python and get your environment up and running and able to run notebooks.
|
||||
|
||||
Take this [Python Learning Path](https://docs.microsoft.com/learn/paths/python-language/?WT.mc_id=academic-77952-leestott), and then get your systems setup by going through these introductory videos:
|
||||
|
||||
https://www.youtube.com/playlist?list=PLlrxD0HtieHhS8VzuMCfQD4uJ9yne1mE6
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
Binary file not shown.
@@ -0,0 +1,150 @@
|
||||
# History of machine learning
|
||||
|
||||
|
||||
> Sketchnote by [Tomomi Imura](https://www.twitter.com/girlie_mac)
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
---
|
||||
|
||||
[](https://youtu.be/N6wxM4wZ7V0 "ML for beginners - History of Machine Learning")
|
||||
|
||||
> 🎥 Click the image above for a short video working through this lesson.
|
||||
|
||||
In this lesson, we will walk through the major milestones in the history of machine learning and artificial intelligence.
|
||||
|
||||
The history of artificial intelligence (AI) as a field is intertwined with the history of machine learning, as the algorithms and computational advances that underpin ML fed into the development of AI. It is useful to remember that, while these fields as distinct areas of inquiry began to crystallize in the 1950s, important [algorithmic, statistical, mathematical, computational and technical discoveries](https://wikipedia.org/wiki/Timeline_of_machine_learning) predated and overlapped this era. In fact, people have been thinking about these questions for [hundreds of years](https://wikipedia.org/wiki/History_of_artificial_intelligence): this article discusses the historical intellectual underpinnings of the idea of a 'thinking machine.'
|
||||
|
||||
---
|
||||
## Notable discoveries
|
||||
|
||||
- 1763, 1812 [Bayes Theorem](https://wikipedia.org/wiki/Bayes%27_theorem) and its predecessors. This theorem and its applications underlie inference, describing the probability of an event occurring based on prior knowledge.
|
||||
- 1805 [Least Square Theory](https://wikipedia.org/wiki/Least_squares) by French mathematician Adrien-Marie Legendre. This theory, which you will learn about in our Regression unit, helps in data fitting.
|
||||
- 1913 [Markov Chains](https://wikipedia.org/wiki/Markov_chain), named after Russian mathematician Andrey Markov, is used to describe a sequence of possible events based on a previous state.
|
||||
- 1957 [Perceptron](https://wikipedia.org/wiki/Perceptron) is a type of linear classifier invented by American psychologist Frank Rosenblatt that underlies advances in deep learning.
|
||||
|
||||
---
|
||||
|
||||
- 1967 [Nearest Neighbor](https://wikipedia.org/wiki/Nearest_neighbor) is an algorithm originally designed to map routes. In an ML context it is used to detect patterns.
|
||||
- 1970 [Backpropagation](https://wikipedia.org/wiki/Backpropagation) is used to train [feedforward neural networks](https://wikipedia.org/wiki/Feedforward_neural_network).
|
||||
- 1982 [Recurrent Neural Networks](https://wikipedia.org/wiki/Recurrent_neural_network) are artificial neural networks derived from feedforward neural networks that create temporal graphs.
|
||||
|
||||
✅ Do a little research. What other dates stand out as pivotal in the history of ML and AI?
|
||||
|
||||
---
|
||||
## 1950: Machines that think
|
||||
|
||||
Alan Turing, a truly remarkable person who was voted [by the public in 2019](https://wikipedia.org/wiki/Icons:_The_Greatest_Person_of_the_20th_Century) as the greatest scientist of the 20th century, is credited as helping to lay the foundation for the concept of a 'machine that can think.' He grappled with naysayers and his own need for empirical evidence of this concept in part by creating the [Turing Test](https://www.bbc.com/news/technology-18475646), which you will explore in our NLP lessons.
|
||||
|
||||
---
|
||||
## 1956: Dartmouth Summer Research Project
|
||||
|
||||
"The Dartmouth Summer Research Project on artificial intelligence was a seminal event for artificial intelligence as a field," and it was here that the term 'artificial intelligence' was coined ([source](https://250.dartmouth.edu/highlights/artificial-intelligence-ai-coined-dartmouth)).
|
||||
|
||||
> Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.
|
||||
|
||||
---
|
||||
|
||||
The lead researcher, mathematics professor John McCarthy, hoped "to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it." The participants included another luminary in the field, Marvin Minsky.
|
||||
|
||||
The workshop is credited with having initiated and encouraged several discussions including "the rise of symbolic methods, systems focussed on limited domains (early expert systems), and deductive systems versus inductive systems." ([source](https://wikipedia.org/wiki/Dartmouth_workshop)).
|
||||
|
||||
---
|
||||
## 1956 - 1974: "The golden years"
|
||||
|
||||
From the 1950s through the mid '70s, optimism ran high in the hope that AI could solve many problems. In 1967, Marvin Minsky stated confidently that "Within a generation ... the problem of creating 'artificial intelligence' will substantially be solved." (Minsky, Marvin (1967), Computation: Finite and Infinite Machines, Englewood Cliffs, N.J.: Prentice-Hall)
|
||||
|
||||
natural language processing research flourished, search was refined and made more powerful, and the concept of 'micro-worlds' was created, where simple tasks were completed using plain language instructions.
|
||||
|
||||
---
|
||||
|
||||
Research was well funded by government agencies, advances were made in computation and algorithms, and prototypes of intelligent machines were built. Some of these machines include:
|
||||
|
||||
* [Shakey the robot](https://wikipedia.org/wiki/Shakey_the_robot), who could maneuver and decide how to perform tasks 'intelligently'.
|
||||
|
||||
|
||||
> Shakey in 1972
|
||||
|
||||
---
|
||||
|
||||
* Eliza, an early 'chatterbot', could converse with people and act as a primitive 'therapist'. You'll learn more about Eliza in the NLP lessons.
|
||||
|
||||
|
||||
> A version of Eliza, a chatbot
|
||||
|
||||
---
|
||||
|
||||
* "Blocks world" was an example of a micro-world where blocks could be stacked and sorted, and experiments in teaching machines to make decisions could be tested. Advances built with libraries such as [SHRDLU](https://wikipedia.org/wiki/SHRDLU) helped propel language processing forward.
|
||||
|
||||
[](https://www.youtube.com/watch?v=QAJz4YKUwqw "blocks world with SHRDLU")
|
||||
|
||||
> 🎥 Click the image above for a video: Blocks world with SHRDLU
|
||||
|
||||
---
|
||||
## 1974 - 1980: "AI Winter"
|
||||
|
||||
By the mid 1970s, it had become apparent that the complexity of making 'intelligent machines' had been understated and that its promise, given the available compute power, had been overblown. Funding dried up and confidence in the field slowed. Some issues that impacted confidence included:
|
||||
---
|
||||
- **Limitations**. Compute power was too limited.
|
||||
- **Combinatorial explosion**. The amount of parameters needed to be trained grew exponentially as more was asked of computers, without a parallel evolution of compute power and capability.
|
||||
- **Paucity of data**. There was a paucity of data that hindered the process of testing, developing, and refining algorithms.
|
||||
- **Are we asking the right questions?**. The very questions that were being asked began to be questioned. Researchers began to field criticism about their approaches:
|
||||
- Turing tests came into question by means, among other ideas, of the 'chinese room theory' which posited that, "programming a digital computer may make it appear to understand language but could not produce real understanding." ([source](https://plato.stanford.edu/entries/chinese-room/))
|
||||
- The ethics of introducing artificial intelligences such as the "therapist" ELIZA into society was challenged.
|
||||
|
||||
---
|
||||
|
||||
At the same time, various AI schools of thought began to form. A dichotomy was established between ["scruffy" vs. "neat AI"](https://wikipedia.org/wiki/Neats_and_scruffies) practices. _Scruffy_ labs tweaked programs for hours until they had the desired results. _Neat_ labs "focused on logic and formal problem solving". ELIZA and SHRDLU were well-known _scruffy_ systems. In the 1980s, as demand emerged to make ML systems reproducible, the _neat_ approach gradually took the forefront as its results are more explainable.
|
||||
|
||||
---
|
||||
## 1980s Expert systems
|
||||
|
||||
As the field grew, its benefit to business became clearer, and in the 1980s so did the proliferation of 'expert systems'. "Expert systems were among the first truly successful forms of artificial intelligence (AI) software." ([source](https://wikipedia.org/wiki/Expert_system)).
|
||||
|
||||
This type of system is actually _hybrid_, consisting partially of a rules engine defining business requirements, and an inference engine that leveraged the rules system to deduce new facts.
|
||||
|
||||
This era also saw increasing attention paid to neural networks.
|
||||
|
||||
---
|
||||
## 1987 - 1993: AI 'Chill'
|
||||
|
||||
The proliferation of specialized expert systems hardware had the unfortunate effect of becoming too specialized. The rise of personal computers also competed with these large, specialized, centralized systems. The democratization of computing had begun, and it eventually paved the way for the modern explosion of big data.
|
||||
|
||||
---
|
||||
## 1993 - 2011
|
||||
|
||||
This epoch saw a new era for ML and AI to be able to solve some of the problems that had been caused earlier by the lack of data and compute power. The amount of data began to rapidly increase and become more widely available, for better and for worse, especially with the advent of the smartphone around 2007. Compute power expanded exponentially, and algorithms evolved alongside. The field began to gain maturity as the freewheeling days of the past began to crystallize into a true discipline.
|
||||
|
||||
---
|
||||
## Now
|
||||
|
||||
Today machine learning and AI touch almost every part of our lives. This era calls for careful understanding of the risks and potentials effects of these algorithms on human lives. As Microsoft's Brad Smith has stated, "Information technology raises issues that go to the heart of fundamental human-rights protections like privacy and freedom of expression. These issues heighten responsibility for tech companies that create these products. In our view, they also call for thoughtful government regulation and for the development of norms around acceptable uses" ([source](https://www.technologyreview.com/2019/12/18/102365/the-future-of-ais-impact-on-society/)).
|
||||
|
||||
---
|
||||
|
||||
It remains to be seen what the future holds, but it is important to understand these computer systems and the software and algorithms that they run. We hope that this curriculum will help you to gain a better understanding so that you can decide for yourself.
|
||||
|
||||
[](https://www.youtube.com/watch?v=mTtDfKgLm54 "The history of deep learning")
|
||||
> 🎥 Click the image above for a video: Yann LeCun discusses the history of deep learning in this lecture
|
||||
|
||||
---
|
||||
## 🚀Challenge
|
||||
|
||||
Dig into one of these historical moments and learn more about the people behind them. There are fascinating characters, and no scientific discovery was ever created in a cultural vacuum. What do you discover?
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
---
|
||||
## Review & Self Study
|
||||
|
||||
Here are items to watch and listen to:
|
||||
|
||||
[This podcast where Amy Boyd discusses the evolution of AI](http://runasradio.com/Shows/Show/739)
|
||||
|
||||
[](https://www.youtube.com/watch?v=EJt3_bFYKss "The history of AI by Amy Boyd")
|
||||
|
||||
---
|
||||
|
||||
## Assignment
|
||||
|
||||
[Create a timeline](assignment.md)
|
||||
@@ -0,0 +1,11 @@
|
||||
# Create a timeline
|
||||
|
||||
## Instructions
|
||||
|
||||
Using [this repo](https://github.com/Digital-Humanities-Toolkit/timeline-builder), create a timeline of some aspect of the history of algorithms, mathematics, statistics, AI, or ML, or a combination of these. You can focus on one person, one idea, or a long timespan of thought. Make sure to add multimedia elements.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | ------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------- |
|
||||
| | A deployed timeline is presented as a GitHub page | The code is incomplete and not deployed | The timeline is incomplete, not well researched and not deployed |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
@@ -0,0 +1,159 @@
|
||||
# Building Machine Learning solutions with responsible AI
|
||||
|
||||
|
||||
> Sketchnote by [Tomomi Imura](https://www.twitter.com/girlie_mac)
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Introduction
|
||||
|
||||
In this curriculum, you will start to discover how machine learning can and is impacting our everyday lives. Even now, systems and models are involved in daily decision-making tasks, such as health care diagnoses, loan approvals or detecting fraud. So, it is important that these models work well to provide outcomes that are trustworthy. Just as any software application, AI systems are going to miss expectations or have an undesirable outcome. That is why it is essential to be about to understand and explain the behavior of an AI model.
|
||||
|
||||
Imagine what can happen when the data you are using to build these models lacks certain demographics, such as race, gender, political view, religion, or disproportionally represents such demographics. What about when the model’s output is interpreted to favor some demographic? What is the consequence for the application? In addition, what happens when the model has an adverse outcome and is harmful to people? Who is accountable for the AI systems behavior? These are some questions we will explore in this curriculum.
|
||||
|
||||
In this lesson, you will:
|
||||
|
||||
- Raise your awareness of the importance of fairness in machine learning and fairness-related harms.
|
||||
- Become familiar with the practice of exploring outliers and unusual scenarios to ensure reliability and safety
|
||||
- Gain understanding on the need to empower everyone by designing inclusive systems
|
||||
- Explore how vital it is to protect privacy and security of data and people
|
||||
- See the importance of having a glass box approach to explain the behavior of AI models
|
||||
- Be mindful of how accountability is essential to build trust in AI systems
|
||||
|
||||
## Prerequisite
|
||||
|
||||
As a prerequisite, please take the "Responsible AI Principles" Learn Path and watch the video below on the topic:
|
||||
|
||||
Learn more about Responsible AI by following this [Learning Path](https://docs.microsoft.com/learn/modules/responsible-ai-principles/?WT.mc_id=academic-77952-leestott)
|
||||
|
||||
[](https://youtu.be/dnC8-uUZXSc "Microsoft's Approach to Responsible AI")
|
||||
|
||||
> 🎥 Click the image above for a video: Microsoft's Approach to Responsible AI
|
||||
|
||||
## Fairness
|
||||
|
||||
AI systems should treat everyone fairly and avoid affecting similar groups of people in different ways. For example, when AI systems provide guidance on medical treatment, loan applications, or employment, they should make the same recommendations to everyone with similar symptoms, financial circumstances, or professional qualifications. Each of us as humans carries around inherited biases that affect our decisions and actions. These biases can be evident in the data that we use to train AI systems. Such manipulation can sometimes happen unintentionally. It is often difficult to consciously know when you are introducing bias in data.
|
||||
|
||||
**“Unfairness”** encompasses negative impacts, or “harms”, for a group of people, such as those defined in terms of race, gender, age, or disability status. The main fairness-related harms can be classified as:
|
||||
|
||||
- **Allocation**, if a gender or ethnicity for example is favored over another.
|
||||
- **Quality of service**. If you train the data for one specific scenario but reality is much more complex, it leads to a poor performing service. For instance, a hand soap dispenser that could not seem to be able to sense people with dark skin. [Reference](https://gizmodo.com/why-cant-this-soap-dispenser-identify-dark-skin-1797931773)
|
||||
- **Denigration**. To unfairly criticize and label something or someone. For example, an image labeling technology infamously mislabeled images of dark-skinned people as gorillas.
|
||||
- **Over- or under- representation**. The idea is that a certain group is not seen in a certain profession, and any service or function that keeps promoting that is contributing to harm.
|
||||
- **Stereotyping**. Associating a given group with pre-assigned attributes. For example, a language translation system betweem English and Turkish may have inaccuraces due to words with stereotypical associations to gender.
|
||||
|
||||
|
||||
> translation to Turkish
|
||||
|
||||
|
||||
> translation back to English
|
||||
|
||||
When designing and testing AI systems, we need to ensure that AI is fair and not programmed to make biased or discriminatory decisions, which human beings are also prohibited from making. Guaranteeing fairness in AI and machine learning remains a complex sociotechnical challenge.
|
||||
|
||||
### Reliability and safety
|
||||
|
||||
To build trust, AI systems need to be reliable, safe, and consistent under normal and unexpected conditions. It is important to know how AI systems will behavior in a variety of situations, especially when they are outliers. When building AI solutions, there needs to be a substantial amount of focus on how to handle a wide variety of circumstances that the AI solutions would encounter. For example, a self-driving car needs to put people's safety as a top priority. As a result, the AI powering the car need to consider all the possible scenarios that the car could come across such as night, thunderstorms or blizzards, kids running across the street, pets, road constructions etc. How well an AI system can handle a wild range of conditions reliably and safely reflects the level of anticipation the data scientist or AI developer considered during the design or testing of the system.
|
||||
|
||||
> [🎥 Click the here for a video: ](https://www.microsoft.com/videoplayer/embed/RE4vvIl)
|
||||
|
||||
### Inclusiveness
|
||||
|
||||
AI systems should be designed to engage and empower everyone. When designing and implementing AI systems data scientists and AI developers identify and address potential barriers in the system that could unintentionally exclude people. For example, there are 1 billion people with disabilities around the world. With the advancement of AI, they can access a wide range of information and opportunities more easily in their daily lives. By addressing the barriers, it creates opportunities to innovate and develop AI products with better experiences that benefit everyone.
|
||||
|
||||
> [🎥 Click the here for a video: inclusiveness in AI](https://www.microsoft.com/videoplayer/embed/RE4vl9v)
|
||||
|
||||
### Security and privacy
|
||||
|
||||
AI systems should be safe and respect people’s privacy. People have less trust in systems that put their privacy, information, or lives at risk. When training machine learning models, we rely on data to produce the best results. In doing so, the origin of the data and integrity must be considered. For example, was the data user submitted or publicly available? Next, while working with the data, it is crucial to develop AI systems that can protect confidential information and resist attacks. As AI becomes more prevalent, protecting privacy and securing important personal and business information is becoming more critical and complex. Privacy and data security issues require especially close attention for AI because access to data is essential for AI systems to make accurate and informed predictions and decisions about people.
|
||||
|
||||
> [🎥 Click the here for a video: security in AI](https://www.microsoft.com/videoplayer/embed/RE4voJF)
|
||||
|
||||
- As an industry we have made significant advancements in Privacy & security, fueled significantly by regulations like the GDPR (General Data Protection Regulation).
|
||||
- Yet with AI systems we must acknowledge the tension between the need for more personal data to make systems more personal and effective – and privacy.
|
||||
- Just like with the birth of connected computers with the internet, we are also seeing a huge uptick in the number of security issues related to AI.
|
||||
- At the same time, we have seen AI being used to improve security. As an example, most modern anti-virus scanners are driven by AI heuristics today.
|
||||
- We need to ensure that our Data Science processes blend harmoniously with the latest privacy and security practices.
|
||||
|
||||
|
||||
### Transparency
|
||||
AI systems should be understandable. A crucial part of transparency is explaining the behavior of AI systems and their components. Improving the understanding of AI systems requires that stakeholders comprehend how and why they function so that they can identify potential performance issues, safety and privacy concerns, biases, exclusionary practices, or unintended outcomes. We also believe that those who use AI systems should be honest and forthcoming about when, why, and how they choose to deploy them. As well as the limitations of the systems they use. For example, if a bank uses an AI system to support its consumer lending decisions, it is important to examine the outcomes and understand which data influences the system’s recommendations. Governments are starting to regulate AI across industries, so data scientists and organizations must explain if an AI system meets regulatory requirements, especially when there is an undesirable outcome.
|
||||
|
||||
> [🎥 Click the here for a video: transparency in AI](https://www.microsoft.com/videoplayer/embed/RE4voJF)
|
||||
|
||||
- Because AI systems are so complex, it is hard to understand how they work and interpret the results.
|
||||
- This lack of understanding affects the way these systems are managed, operationalized, and documented.
|
||||
- This lack of understanding more importantly affects the decisions made using the results these systems produce.
|
||||
|
||||
### Accountability
|
||||
|
||||
The people who design and deploy AI systems must be accountable for how their systems operate. The need for accountability is particularly crucial with sensitive use technologies like facial recognition. Recently, there has been a growing demand for facial recognition technology, especially from law enforcement organizations who see the potential of the technology in uses like finding missing children. However, these technologies could potentially be used by a government to put their citizens’ fundamental freedoms at risk by, for example, enabling continuous surveillance of specific individuals. Hence, data scientists and organizations need to be responsible for how their AI system impacts individuals or society.
|
||||
|
||||
[](https://www.youtube.com/watch?v=Wldt8P5V6D0 "Microsoft's Approach to Responsible AI")
|
||||
|
||||
> 🎥 Click the image above for a video: Warnings of Mass Surveillance Through Facial Recognition
|
||||
|
||||
Ultimately one of the biggest questions for our generation, as the first generation that is bringing AI to society, is how to ensure that computers will remain accountable to people and how to ensure that the people that design computers remain accountable to everyone else.
|
||||
|
||||
## Impact assessment
|
||||
|
||||
Before training a machine learning model, it is important to conduct an impact assessmet to understand the purpose of the AI system; what the intended use is; where it will be deployed; and who will be interacting with the system. These are helpful for reviewer(s) or testers evaluating the system to know what factors to take into consideration when identifying potential risks and expected consequences.
|
||||
|
||||
The following are areas of focus when conducting an impact assessment:
|
||||
|
||||
* **Adverse impact on individuals**. Being aware of any restriction or requirements, unsupported use or any known limitations hindering the system's performance is vital to ensure that the system is not used in a way that could cause harm to individuals.
|
||||
* **Data requirements**. Gaining an understanding of how and where the system will use data enables reviewers to explore any data requirements you would need to be mindful of (e.g., GDPR or HIPAA data regulations). In addition, examine whether the source or quantity of data is substantial for training.
|
||||
* **Summary of impact**. Gather a list of potential harms that could arise from using the system. Throughout the ML lifecycle, review if the issues identified are mitigated or addressed.
|
||||
* **Applicable goals** for each of the six core principles. Assess if the goals from each of the principles are met and if there are any gaps.
|
||||
|
||||
|
||||
## Debugging with responsible AI
|
||||
|
||||
Similar to debugging a software application, debugging an AI system is a necessary process of identifying and resolving issues in the system. There are many factors that would affect a model not performing as expected or responsibly. Most traditional model performance metrics are quantitative aggregates of a model's performance, which are not sufficient to analyze how a model violates the responsible AI principles. Furthermore, a machine learning model is a black box that makes it difficult to understand what drives its outcome or provide explanation when it makes a mistake. Later in this course, we will learn how to use the Responsible AI dashboard to help debug AI systems. The dashboard provides a holistic tool for data scientists and AI developers to perform:
|
||||
|
||||
* **Error analysis**. To identify the error distribution of the model that can affect the system's fairness or reliability.
|
||||
* **Model overview**. To discover where there are disparities in the model's performance across data cohorts.
|
||||
* **Data analysis**. To understand the data distribution and identify any potential bias in the data that could lead to fairness, inclusiveness, and reliability issues.
|
||||
* **Model interpretability**. To understand what affects or influences the model's predictions. This helps in explaining the model's behavior, which is important for transparency and accountability.
|
||||
|
||||
|
||||
## 🚀 Challenge
|
||||
|
||||
To prevent harms from being introduced in the first place, we should:
|
||||
|
||||
- have a diversity of backgrounds and perspectives among the people working on systems
|
||||
- invest in datasets that reflect the diversity of our society
|
||||
- develop better methods throughout the machine learning lifecycle for detecting and correcting responible AI when it occurs
|
||||
|
||||
Think about real-life scenarios where a model's untrustworthiness is evident in model-building and usage. What else should we consider?
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
In this lesson, you have learned some basics of the concepts of fairness and unfairness in machine learning.
|
||||
|
||||
Watch this workshop to dive deeper into the topics:
|
||||
|
||||
- In pursuit of responsible AI: Bringing principles to practice by Besmira Nushi, Mehrnoosh Sameki and Amit Sharma
|
||||
|
||||
[](https://www.youtube.com/watch?v=tGgJCrA-MZU "RAI Toolbox: An open-source framework for building responsible AI")
|
||||
|
||||
> 🎥 Click the image above for a video: RAI Toolbox: An open-source framework for building responsible AI by Besmira Nushi, Mehrnoosh Sameki, and Amit Sharma
|
||||
|
||||
Also, read:
|
||||
|
||||
- Microsoft’s RAI resource center: [Responsible AI Resources – Microsoft AI](https://www.microsoft.com/ai/responsible-ai-resources?activetab=pivot1%3aprimaryr4)
|
||||
|
||||
- Microsoft’s FATE research group: [FATE: Fairness, Accountability, Transparency, and Ethics in AI - Microsoft Research](https://www.microsoft.com/research/theme/fate/)
|
||||
|
||||
RAI Toolbox:
|
||||
|
||||
- [Responsible AI Toolbox GitHub repository](https://github.com/microsoft/responsible-ai-toolbox)
|
||||
|
||||
Read about Azure Machine Learning's tools to ensure fairness:
|
||||
|
||||
- [Azure Machine Learning](https://docs.microsoft.com/azure/machine-learning/concept-fairness-ml?WT.mc_id=academic-77952-leestott)
|
||||
|
||||
## Assignment
|
||||
|
||||
[Explore RAI Toolbox](assignment.md)
|
||||
@@ -0,0 +1,11 @@
|
||||
# Explore the Responsible AI Toolbox
|
||||
|
||||
## Instructions
|
||||
|
||||
In this lesson you learned about the Responsible AI Toolbox, an "open-source, community-driven project to help data scientists to analyze and improve AI systems." For this assignment, explore one of RAI Toolbox's [notebooks](https://github.com/microsoft/responsible-ai-toolbox/blob/main/notebooks/responsibleaidashboard/tabular/getting-started.ipynb) and report your findings in a paper or presentation.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | --------- | -------- | ----------------- |
|
||||
| | A paper or powerpoint presentation is presented discussing Fairlearn's systems, the notebook that was run, and the conclusions drawn from running it | A paper is presented without conclusions | No paper is presented |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 424 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.2 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 259 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
@@ -0,0 +1,118 @@
|
||||
# Techniques of Machine Learning
|
||||
|
||||
The process of building, using, and maintaining machine learning models and the data they use is a very different process from many other development workflows. In this lesson, we will demystify the process, and outline the main techniques you need to know. You will:
|
||||
|
||||
- Understand the processes underpinning machine learning at a high level.
|
||||
- Explore base concepts such as 'models', 'predictions', and 'training data'.
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
[](https://youtu.be/4NGM0U2ZSHU "ML for beginners - Techniques of Machine Learning")
|
||||
|
||||
> 🎥 Click the image above for a short video working through this lesson.
|
||||
|
||||
## Introduction
|
||||
|
||||
On a high level, the craft of creating machine learning (ML) processes is comprised of a number of steps:
|
||||
|
||||
1. **Decide on the question**. Most ML processes start by asking a question that cannot be answered by a simple conditional program or rules-based engine. These questions often revolve around predictions based on a collection of data.
|
||||
2. **Collect and prepare data**. To be able to answer your question, you need data. The quality and, sometimes, quantity of your data will determine how well you can answer your initial question. Visualizing data is an important aspect of this phase. This phase also includes splitting the data into a training and testing group to build a model.
|
||||
3. **Choose a training method**. Depending on your question and the nature of your data, you need to choose how you want to train a model to best reflect your data and make accurate predictions against it. This is the part of your ML process that requires specific expertise and, often, a considerable amount of experimentation.
|
||||
4. **Train the model**. Using your training data, you'll use various algorithms to train a model to recognize patterns in the data. The model might leverage internal weights that can be adjusted to privilege certain parts of the data over others to build a better model.
|
||||
5. **Evaluate the model**. You use never before seen data (your testing data) from your collected set to see how the model is performing.
|
||||
6. **Parameter tuning**. Based on the performance of your model, you can redo the process using different parameters, or variables, that control the behavior of the algorithms used to train the model.
|
||||
7. **Predict**. Use new inputs to test the accuracy of your model.
|
||||
|
||||
## What question to ask
|
||||
|
||||
Computers are particularly skilled at discovering hidden patterns in data. This utility is very helpful for researchers who have questions about a given domain that cannot be easily answered by creating a conditionally-based rules engine. Given an actuarial task, for example, a data scientist might be able to construct handcrafted rules around the mortality of smokers vs non-smokers.
|
||||
|
||||
When many other variables are brought into the equation, however, a ML model might prove more efficient to predict future mortality rates based on past health history. A more cheerful example might be making weather predictions for the month of April in a given location based on data that includes latitude, longitude, climate change, proximity to the ocean, patterns of the jet stream, and more.
|
||||
|
||||
✅ This [slide deck](https://www2.cisl.ucar.edu/sites/default/files/2021-10/0900%20June%2024%20Haupt_0.pdf) on weather models offers a historical perspective for using ML in weather analysis.
|
||||
|
||||
## Pre-building tasks
|
||||
|
||||
Before starting to build your model, there are several tasks you need to complete. To test your question and form a hypothesis based on a model's predictions, you need to identify and configure several elements.
|
||||
|
||||
### Data
|
||||
|
||||
To be able to answer your question with any kind of certainty, you need a good amount of data of the right type. There are two things you need to do at this point:
|
||||
|
||||
- **Collect data**. Keeping in mind the previous lesson on fairness in data analysis, collect your data with care. Be aware of the sources of this data, any inherent biases it might have, and document its origin.
|
||||
- **Prepare data**. There are several steps in the data preparation process. You might need to collate data and normalize it if it comes from diverse sources. You can improve the data's quality and quantity through various methods such as converting strings to numbers (as we do in [Clustering](../../5-Clustering/1-Visualize/README.md)). You might also generate new data, based on the original (as we do in [Classification](../../4-Classification/1-Introduction/README.md)). You can clean and edit the data (as we will prior to the [Web App](../../3-Web-App/README.md) lesson). Finally, you might also need to randomize it and shuffle it, depending on your training techniques.
|
||||
|
||||
✅ After collecting and processing your data, take a moment to see if its shape will allow you to address your intended question. It may be that the data will not perform well in your given task, as we discover in our [Clustering](../../5-Clustering/1-Visualize/README.md) lessons!
|
||||
|
||||
### Features and Target
|
||||
|
||||
A [feature](https://www.datasciencecentral.com/profiles/blogs/an-introduction-to-variable-and-feature-selection) is a measurable property of your data. In many datasets it is expressed as a column heading like 'date' 'size' or 'color'. Your feature variable, usually represented as `X` in code, represents the input variable which will be used to train a model.
|
||||
|
||||
A target is a thing you are trying to predict. Target, usually represented as `y` in code, represents the answer to the question you are trying to ask of your data: in December, what **color** pumpkins will be cheapest? in San Francisco, what neighborhoods will have the best real estate **price**? Sometimes target is also referred to as a label attribute.
|
||||
|
||||
### Selecting your feature variable
|
||||
|
||||
🎓 **Feature Selection and Feature Extraction** How do you know which variable to choose when building a model? You'll probably go through a process of feature selection or feature extraction to choose the right variables for the most performant model. They're not the same thing, however: "Feature extraction creates new features from functions of the original features, whereas feature selection returns a subset of the features." ([source](https://wikipedia.org/wiki/Feature_selection))
|
||||
|
||||
### Visualize your data
|
||||
|
||||
An important aspect of the data scientist's toolkit is the power to visualize data using several excellent libraries such as Seaborn or MatPlotLib. Representing your data visually might allow you to uncover hidden correlations that you can leverage. Your visualizations might also help you to uncover bias or unbalanced data (as we discover in [Classification](../../4-Classification/2-Classifiers-1/README.md)).
|
||||
|
||||
### Split your dataset
|
||||
|
||||
Prior to training, you need to split your dataset into two or more parts of unequal size that still represent the data well.
|
||||
|
||||
- **Training**. This part of the dataset is fit to your model to train it. This set constitutes the majority of the original dataset.
|
||||
- **Testing**. A test dataset is an independent group of data, often gathered from the original data, that you use to confirm the performance of the built model.
|
||||
- **Validating**. A validation set is a smaller independent group of examples that you use to tune the model's hyperparameters, or architecture, to improve the model. Depending on your data's size and the question you are asking, you might not need to build this third set (as we note in [Time Series Forecasting](../../7-TimeSeries/1-Introduction/README.md)).
|
||||
|
||||
## Building a model
|
||||
|
||||
Using your training data, your goal is to build a model, or a statistical representation of your data, using various algorithms to **train** it. Training a model exposes it to data and allows it to make assumptions about perceived patterns it discovers, validates, and accepts or rejects.
|
||||
|
||||
### Decide on a training method
|
||||
|
||||
Depending on your question and the nature of your data, you will choose a method to train it. Stepping through [Scikit-learn's documentation](https://scikit-learn.org/stable/user_guide.html) - which we use in this course - you can explore many ways to train a model. Depending on your experience, you might have to try several different methods to build the best model. You are likely to go through a process whereby data scientists evaluate the performance of a model by feeding it unseen data, checking for accuracy, bias, and other quality-degrading issues, and selecting the most appropriate training method for the task at hand.
|
||||
|
||||
### Train a model
|
||||
|
||||
Armed with your training data, you are ready to 'fit' it to create a model. You will notice that in many ML libraries you will find the code 'model.fit' - it is at this time that you send in your feature variable as an array of values (usually 'X') and a target variable (usually 'y').
|
||||
|
||||
### Evaluate the model
|
||||
|
||||
Once the training process is complete (it can take many iterations, or 'epochs', to train a large model), you will be able to evaluate the model's quality by using test data to gauge its performance. This data is a subset of the original data that the model has not previously analyzed. You can print out a table of metrics about your model's quality.
|
||||
|
||||
🎓 **Model fitting**
|
||||
|
||||
In the context of machine learning, model fitting refers to the accuracy of the model's underlying function as it attempts to analyze data with which it is not familiar.
|
||||
|
||||
🎓 **Underfitting** and **overfitting** are common problems that degrade the quality of the model, as the model fits either not well enough or too well. This causes the model to make predictions either too closely aligned or too loosely aligned with its training data. An overfit model predicts training data too well because it has learned the data's details and noise too well. An underfit model is not accurate as it can neither accurately analyze its training data nor data it has not yet 'seen'.
|
||||
|
||||
|
||||
> Infographic by [Jen Looper](https://twitter.com/jenlooper)
|
||||
|
||||
## Parameter tuning
|
||||
|
||||
Once your initial training is complete, observe the quality of the model and consider improving it by tweaking its 'hyperparameters'. Read more about the process [in the documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters?WT.mc_id=academic-77952-leestott).
|
||||
|
||||
## Prediction
|
||||
|
||||
This is the moment where you can use completely new data to test your model's accuracy. In an 'applied' ML setting, where you are building web assets to use the model in production, this process might involve gathering user input (a button press, for example) to set a variable and send it to the model for inference, or evaluation.
|
||||
|
||||
In these lessons, you will discover how to use these steps to prepare, build, test, evaluate, and predict - all the gestures of a data scientist and more, as you progress in your journey to become a 'full stack' ML engineer.
|
||||
|
||||
---
|
||||
|
||||
## 🚀Challenge
|
||||
|
||||
Draw a flow chart reflecting the steps of a ML practitioner. Where do you see yourself right now in the process? Where do you predict you will find difficulty? What seems easy to you?
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
Search online for interviews with data scientists who discuss their daily work. Here is [one](https://www.youtube.com/watch?v=Z3IjgbbCEfs).
|
||||
|
||||
## Assignment
|
||||
|
||||
[Interview a data scientist](assignment.md)
|
||||
@@ -0,0 +1,11 @@
|
||||
# Interview a data scientist
|
||||
|
||||
## Instructions
|
||||
|
||||
In your company, in a user group, or among your friends or fellow students, talk to someone who works professionally as a data scientist. Write a short paper (500 words) about their daily occupations. Are they specialists, or do they work 'full stack'?
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | --------------------- |
|
||||
| | An essay of the correct length, with attributed sources, is presented as a .doc file | The essay is poorly attributed or shorter than the required length | No essay is presented |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
@@ -0,0 +1,22 @@
|
||||
# Introduction to machine learning
|
||||
|
||||
In this section of the curriculum, you will be introduced to the fundamental concepts of machine learning, including what it is, its history, and the techniques researchers use to apply it in real-world scenarios. Let's explore this exciting world of ML together!
|
||||
|
||||
|
||||
> Photo by <a href="https://unsplash.com/@bill_oxford?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Bill Oxford</a> on <a href="https://unsplash.com/s/photos/globe?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
|
||||
|
||||
### Lessons
|
||||
|
||||
1. [Introduction to machine learning](1-intro-to-ML/README.md)
|
||||
1. [The History of machine learning and AI](2-history-of-ML/README.md)
|
||||
1. [Fairness and machine learning](3-fairness/README.md)
|
||||
1. [Techniques of machine learning](4-techniques-of-ML/README.md)
|
||||
### Credits
|
||||
|
||||
"Introduction to Machine Learning" was written with ♥️ by a team of folks including [Muhammad Sakib Khan Inan](https://twitter.com/Sakibinan), [Ornella Altunyan](https://twitter.com/ornelladotcom) and [Jen Looper](https://twitter.com/jenlooper)
|
||||
|
||||
"The History of Machine Learning" was written with ♥️ by [Jen Looper](https://twitter.com/jenlooper) and [Amy Boyd](https://twitter.com/AmyKateNicho)
|
||||
|
||||
"Fairness and Machine Learning" was written with ♥️ by [Tomomi Imura](https://twitter.com/girliemac)
|
||||
|
||||
"Techniques of Machine Learning" was written with ♥️ by [Jen Looper](https://twitter.com/jenlooper) and [Chris Noring](https://twitter.com/softchris)
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 348 KiB |
@@ -0,0 +1,226 @@
|
||||
# Get started with Python and Scikit-learn for regression models
|
||||
|
||||
|
||||
|
||||
> Sketchnote by [Tomomi Imura](https://www.twitter.com/girlie_mac)
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
> ### [This lesson is available in R!](./solution/R/lesson_1.html)
|
||||
|
||||
## Introduction
|
||||
|
||||
In these four lessons, you will discover how to build regression models. We will discuss what these are for shortly. But before you do anything, make sure you have the right tools in place to start the process!
|
||||
|
||||
In this lesson, you will learn how to:
|
||||
|
||||
- Configure your computer for local machine learning tasks.
|
||||
- Work with Jupyter Notebooks.
|
||||
- Use Scikit-learn, including installation.
|
||||
- Explore linear regression with a hands-on exercise.
|
||||
|
||||
## Installations and configurations
|
||||
|
||||
[](https://youtu.be/-DfeD2k2Kj0 "ML for beginners -Setup your tools ready to build Machine Learning models")
|
||||
|
||||
> 🎥 Click the image above for a short video working through configuring your computer for ML.
|
||||
|
||||
1. **Install Python**. Ensure that [Python](https://www.python.org/downloads/) is installed on your computer. You will use Python for many data science and machine learning tasks. Most computer systems already include a Python installation. There are useful [Python Coding Packs](https://code.visualstudio.com/learn/educators/installers?WT.mc_id=academic-77952-leestott) available as well, to ease the setup for some users.
|
||||
|
||||
Some usages of Python, however, require one version of the software, whereas others require a different version. For this reason, it's useful to work within a [virtual environment](https://docs.python.org/3/library/venv.html).
|
||||
|
||||
2. **Install Visual Studio Code**. Make sure you have Visual Studio Code installed on your computer. Follow these instructions to [install Visual Studio Code](https://code.visualstudio.com/) for the basic installation. You are going to use Python in Visual Studio Code in this course, so you might want to brush up on how to [configure Visual Studio Code](https://docs.microsoft.com/learn/modules/python-install-vscode?WT.mc_id=academic-77952-leestott) for Python development.
|
||||
|
||||
> Get comfortable with Python by working through this collection of [Learn modules](https://docs.microsoft.com/users/jenlooper-2911/collections/mp1pagggd5qrq7?WT.mc_id=academic-77952-leestott)
|
||||
>
|
||||
> [](https://youtu.be/yyQM70vi7V8 "Setup Python with Visual Studio Code")
|
||||
>
|
||||
> 🎥 Click the image above for a video: using Python within VS Code.
|
||||
|
||||
3. **Install Scikit-learn**, by following [these instructions](https://scikit-learn.org/stable/install.html). Since you need to ensure that you use Python 3, it's recommended that you use a virtual environment. Note, if you are installing this library on a M1 Mac, there are special instructions on the page linked above.
|
||||
|
||||
1. **Install Jupyter Notebook**. You will need to [install the Jupyter package](https://pypi.org/project/jupyter/).
|
||||
|
||||
## Your ML authoring environment
|
||||
|
||||
You are going to use **notebooks** to develop your Python code and create machine learning models. This type of file is a common tool for data scientists, and they can be identified by their suffix or extension `.ipynb`.
|
||||
|
||||
Notebooks are an interactive environment that allow the developer to both code and add notes and write documentation around the code which is quite helpful for experimental or research-oriented projects.
|
||||
|
||||
[](https://youtu.be/7E-jC8FLA2E "ML for beginners - Set up Jupyter Notebooks to start building regression models")
|
||||
|
||||
> 🎥 Click the image above for a short video working through this exercise.
|
||||
|
||||
### Exercise - work with a notebook
|
||||
|
||||
In this folder, you will find the file _notebook.ipynb_.
|
||||
|
||||
1. Open _notebook.ipynb_ in Visual Studio Code.
|
||||
|
||||
A Jupyter server will start with Python 3+ started. You will find areas of the notebook that can be `run`, pieces of code. You can run a code block, by selecting the icon that looks like a play button.
|
||||
|
||||
1. Select the `md` icon and add a bit of markdown, and the following text **# Welcome to your notebook**.
|
||||
|
||||
Next, add some Python code.
|
||||
|
||||
1. Type **print('hello notebook')** in the code block.
|
||||
1. Select the arrow to run the code.
|
||||
|
||||
You should see the printed statement:
|
||||
|
||||
```output
|
||||
hello notebook
|
||||
```
|
||||
|
||||
|
||||
|
||||
You can interleaf your code with comments to self-document the notebook.
|
||||
|
||||
✅ Think for a minute how different a web developer's working environment is versus that of a data scientist.
|
||||
|
||||
## Up and running with Scikit-learn
|
||||
|
||||
Now that Python is set up in your local environment, and you are comfortable with Jupyter Notebooks, let's get equally comfortable with Scikit-learn (pronounce it `sci` as in `science`). Scikit-learn provides an [extensive API](https://scikit-learn.org/stable/modules/classes.html#api-ref) to help you perform ML tasks.
|
||||
|
||||
According to their [website](https://scikit-learn.org/stable/getting_started.html), "Scikit-learn is an open source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection and evaluation, and many other utilities."
|
||||
|
||||
In this course, you will use Scikit-learn and other tools to build machine learning models to perform what we call 'traditional machine learning' tasks. We have deliberately avoided neural networks and deep learning, as they are better covered in our forthcoming 'AI for Beginners' curriculum.
|
||||
|
||||
Scikit-learn makes it straightforward to build models and evaluate them for use. It is primarily focused on using numeric data and contains several ready-made datasets for use as learning tools. It also includes pre-built models for students to try. Let's explore the process of loading prepackaged data and using a built in estimator first ML model with Scikit-learn with some basic data.
|
||||
|
||||
## Exercise - your first Scikit-learn notebook
|
||||
|
||||
> This tutorial was inspired by the [linear regression example](https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py) on Scikit-learn's web site.
|
||||
|
||||
|
||||
[](https://youtu.be/2xkXL5EUpS0 "ML for beginners - Your First Linear Regression Project in Python")
|
||||
|
||||
> 🎥 Click the image above for a short video working through this exercise.
|
||||
|
||||
In the _notebook.ipynb_ file associated to this lesson, clear out all the cells by pressing the 'trash can' icon.
|
||||
|
||||
In this section, you will work with a small dataset about diabetes that is built into Scikit-learn for learning purposes. Imagine that you wanted to test a treatment for diabetic patients. Machine Learning models might help you determine which patients would respond better to the treatment, based on combinations of variables. Even a very basic regression model, when visualized, might show information about variables that would help you organize your theoretical clinical trials.
|
||||
|
||||
✅ There are many types of regression methods, and which one you pick depends on the answer you're looking for. If you want to predict the probable height for a person of a given age, you'd use linear regression, as you're seeking a **numeric value**. If you're interested in discovering whether a type of cuisine should be considered vegan or not, you're looking for a **category assignment** so you would use logistic regression. You'll learn more about logistic regression later. Think a bit about some questions you can ask of data, and which of these methods would be more appropriate.
|
||||
|
||||
Let's get started on this task.
|
||||
|
||||
### Import libraries
|
||||
|
||||
For this task we will import some libraries:
|
||||
|
||||
- **matplotlib**. It's a useful [graphing tool](https://matplotlib.org/) and we will use it to create a line plot.
|
||||
- **numpy**. [numpy](https://numpy.org/doc/stable/user/whatisnumpy.html) is a useful library for handling numeric data in Python.
|
||||
- **sklearn**. This is the [Scikit-learn](https://scikit-learn.org/stable/user_guide.html) library.
|
||||
|
||||
Import some libraries to help with your tasks.
|
||||
|
||||
1. Add imports by typing the following code:
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
from sklearn import datasets, linear_model, model_selection
|
||||
```
|
||||
|
||||
Above you are importing `matplotlib`, `numpy` and you are importing `datasets`, `linear_model` and `model_selection` from `sklearn`. `model_selection` is used for splitting data into training and test sets.
|
||||
|
||||
### The diabetes dataset
|
||||
|
||||
The built-in [diabetes dataset](https://scikit-learn.org/stable/datasets/toy_dataset.html#diabetes-dataset) includes 442 samples of data around diabetes, with 10 feature variables, some of which include:
|
||||
|
||||
- age: age in years
|
||||
- bmi: body mass index
|
||||
- bp: average blood pressure
|
||||
- s1 tc: T-Cells (a type of white blood cells)
|
||||
|
||||
✅ This dataset includes the concept of 'sex' as a feature variable important to research around diabetes. Many medical datasets include this type of binary classification. Think a bit about how categorizations such as this might exclude certain parts of a population from treatments.
|
||||
|
||||
Now, load up the X and y data.
|
||||
|
||||
> 🎓 Remember, this is supervised learning, and we need a named 'y' target.
|
||||
|
||||
In a new code cell, load the diabetes dataset by calling `load_diabetes()`. The input `return_X_y=True` signals that `X` will be a data matrix, and `y` will be the regression target.
|
||||
|
||||
1. Add some print commands to show the shape of the data matrix and its first element:
|
||||
|
||||
```python
|
||||
X, y = datasets.load_diabetes(return_X_y=True)
|
||||
print(X.shape)
|
||||
print(X[0])
|
||||
```
|
||||
|
||||
What you are getting back as a response, is a tuple. What you are doing is to assign the two first values of the tuple to `X` and `y` respectively. Learn more [about tuples](https://wikipedia.org/wiki/Tuple).
|
||||
|
||||
You can see that this data has 442 items shaped in arrays of 10 elements:
|
||||
|
||||
```text
|
||||
(442, 10)
|
||||
[ 0.03807591 0.05068012 0.06169621 0.02187235 -0.0442235 -0.03482076
|
||||
-0.04340085 -0.00259226 0.01990842 -0.01764613]
|
||||
```
|
||||
|
||||
✅ Think a bit about the relationship between the data and the regression target. Linear regression predicts relationships between feature X and target variable y. Can you find the [target](https://scikit-learn.org/stable/datasets/toy_dataset.html#diabetes-dataset) for the diabetes dataset in the documentation? What is this dataset demonstrating, given that target?
|
||||
|
||||
2. Next, select a portion of this dataset to plot by selecting the 3rd column of the dataset. You can do this by using the `:` operator to select all rows, and then selecting the 3rd column using the index (2). You can also reshape the data to be a 2D array - as required for plotting - by using `reshape(n_rows, n_columns)`. If one of the parameter is -1, the corresponding dimension is calculated automatically.
|
||||
|
||||
```python
|
||||
X = X[:, 2]
|
||||
X = X.reshape((-1,1))
|
||||
```
|
||||
|
||||
✅ At any time, print out the data to check its shape.
|
||||
|
||||
3. Now that you have data ready to be plotted, you can see if a machine can help determine a logical split between the numbers in this dataset. To do this, you need to split both the data (X) and the target (y) into test and training sets. Scikit-learn has a straightforward way to do this; you can split your test data at a given point.
|
||||
|
||||
```python
|
||||
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.33)
|
||||
```
|
||||
|
||||
4. Now you are ready to train your model! Load up the linear regression model and train it with your X and y training sets using `model.fit()`:
|
||||
|
||||
```python
|
||||
model = linear_model.LinearRegression()
|
||||
model.fit(X_train, y_train)
|
||||
```
|
||||
|
||||
✅ `model.fit()` is a function you'll see in many ML libraries such as TensorFlow
|
||||
|
||||
5. Then, create a prediction using test data, using the function `predict()`. This will be used to draw the line between data groups
|
||||
|
||||
```python
|
||||
y_pred = model.predict(X_test)
|
||||
```
|
||||
|
||||
6. Now it's time to show the data in a plot. Matplotlib is a very useful tool for this task. Create a scatterplot of all the X and y test data, and use the prediction to draw a line in the most appropriate place, between the model's data groupings.
|
||||
|
||||
```python
|
||||
plt.scatter(X_test, y_test, color='black')
|
||||
plt.plot(X_test, y_pred, color='blue', linewidth=3)
|
||||
plt.xlabel('Scaled BMIs')
|
||||
plt.ylabel('Disease Progression')
|
||||
plt.title('A Graph Plot Showing Diabetes Progression Against BMI')
|
||||
plt.show()
|
||||
```
|
||||
|
||||
|
||||
|
||||
✅ Think a bit about what's going on here. A straight line is running through many small dots of data, but what is it doing exactly? Can you see how you should be able to use this line to predict where a new, unseen data point should fit in relationship to the plot's y axis? Try to put into words the practical use of this model.
|
||||
|
||||
Congratulations, you built your first linear regression model, created a prediction with it, and displayed it in a plot!
|
||||
|
||||
---
|
||||
## 🚀Challenge
|
||||
|
||||
Plot a different variable from this dataset. Hint: edit this line: `X = X[:,2]`. Given this dataset's target, what are you able to discover about the progression of diabetes as a disease?
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
In this tutorial, you worked with simple linear regression, rather than univariate or multiple linear regression. Read a little about the differences between these methods, or take a look at [this video](https://www.coursera.org/lecture/quantifying-relationships-regression-models/linear-vs-nonlinear-categorical-variables-ai2Ef)
|
||||
|
||||
Read more about the concept of regression and think about what kinds of questions can be answered by this technique. Take this [tutorial](https://docs.microsoft.com/learn/modules/train-evaluate-regression-models?WT.mc_id=academic-77952-leestott) to deepen your understanding.
|
||||
|
||||
## Assignment
|
||||
|
||||
[A different dataset](assignment.md)
|
||||
@@ -0,0 +1,13 @@
|
||||
# Regression with Scikit-learn
|
||||
|
||||
## Instructions
|
||||
|
||||
Take a look at the [Linnerud dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_linnerud.html#sklearn.datasets.load_linnerud) in Scikit-learn. This dataset has multiple [targets](https://scikit-learn.org/stable/datasets/toy_dataset.html#linnerrud-dataset): 'It consists of three exercise (data) and three physiological (target) variables collected from twenty middle-aged men in a fitness club'.
|
||||
|
||||
In your own words, describe how to create a Regression model that would plot the relationship between the waistline and how many situps are accomplished. Do the same for the other datapoints in this dataset.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| ------------------------------ | ----------------------------------- | ----------------------------- | -------------------------- |
|
||||
| Submit a descriptive paragraph | Well-written paragraph is submitted | A few sentences are submitted | No description is supplied |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 558 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
@@ -0,0 +1 @@
|
||||
This is a temporary placeholder
|
||||
@@ -0,0 +1,441 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "lesson_1-R.ipynb",
|
||||
"provenance": [],
|
||||
"collapsed_sections": [],
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "ir",
|
||||
"display_name": "R"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "R"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Build a regression model: Get started with R and Tidymodels for regression models"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "YJUHCXqK57yz"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## Introduction to Regression - Lesson 1\n",
|
||||
"\n",
|
||||
"#### Putting it into perspective\n",
|
||||
"\n",
|
||||
"✅ There are many types of regression methods, and which one you pick depends on the answer you're looking for. If you want to predict the probable height for a person of a given age, you'd use `linear regression`, as you're seeking a **numeric value**. If you're interested in discovering whether a type of cuisine should be considered vegan or not, you're looking for a **category assignment** so you would use `logistic regression`. You'll learn more about logistic regression later. Think a bit about some questions you can ask of data, and which of these methods would be more appropriate.\n",
|
||||
"\n",
|
||||
"In this section, you will work with a [small dataset about diabetes](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html). Imagine that you wanted to test a treatment for diabetic patients. Machine Learning models might help you determine which patients would respond better to the treatment, based on combinations of variables. Even a very basic regression model, when visualized, might show information about variables that would help you organize your theoretical clinical trials.\n",
|
||||
"\n",
|
||||
"That said, let's get started on this task!\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/encouRage.jpg\"\n",
|
||||
" width=\"630\"/>\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\n",
|
||||
"\n",
|
||||
"<!--<br>Artwork by @allison_horst-->"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "LWNNzfqd6feZ"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 1. Loading up our tool set\n",
|
||||
"\n",
|
||||
"For this task, we'll require the following packages:\n",
|
||||
"\n",
|
||||
"- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!\n",
|
||||
"\n",
|
||||
"- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.\n",
|
||||
"\n",
|
||||
"You can have them installed as:\n",
|
||||
"\n",
|
||||
"`install.packages(c(\"tidyverse\", \"tidymodels\"))`\n",
|
||||
"\n",
|
||||
"The script below checks whether you have the packages required to complete this module and installs them for you in case some are missing."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "FIo2YhO26wI9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"source": [
|
||||
"suppressWarnings(if(!require(\"pacman\")) install.packages(\"pacman\"))\n",
|
||||
"pacman::p_load(tidyverse, tidymodels)"
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"output_type": "stream",
|
||||
"name": "stderr",
|
||||
"text": [
|
||||
"Loading required package: pacman\n",
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"id": "cIA9fz9v7Dss",
|
||||
"colab": {
|
||||
"base_uri": "https://localhost:8080/"
|
||||
},
|
||||
"outputId": "2df7073b-86b2-4b32-cb86-0da605a0dc11"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now, let's load these awesome packages and make them available in our current R session.(This is for mere illustration, `pacman::p_load()` already did that for you)"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "gpO_P_6f9WUG"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# load the core Tidyverse packages\r\n",
|
||||
"library(tidyverse)\r\n",
|
||||
"\r\n",
|
||||
"# load the core Tidymodels packages\r\n",
|
||||
"library(tidymodels)\r\n"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "NLMycgG-9ezO"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 2. The diabetes dataset\n",
|
||||
"\n",
|
||||
"In this exercise, we'll put our regression skills into display by making predictions on a diabetes dataset. The [diabetes dataset](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.rwrite1.txt) includes `442 samples` of data around diabetes, with 10 predictor feature variables, `age`, `sex`, `body mass index`, `average blood pressure`, and `six blood serum measurements` as well as an outcome variable `y`: a quantitative measure of disease progression one year after baseline.\n",
|
||||
"\n",
|
||||
"|Number of observations|442|\n",
|
||||
"|----------------------|:---|\n",
|
||||
"|Number of predictors|First 10 columns are numeric predictive|\n",
|
||||
"|Outcome/Target|Column 11 is a quantitative measure of disease progression one year after baseline|\n",
|
||||
"|Predictor Information|- age in years\n",
|
||||
"||- sex\n",
|
||||
"||- bmi body mass index\n",
|
||||
"||- bp average blood pressure\n",
|
||||
"||- s1 tc, total serum cholesterol\n",
|
||||
"||- s2 ldl, low-density lipoproteins\n",
|
||||
"||- s3 hdl, high-density lipoproteins\n",
|
||||
"||- s4 tch, total cholesterol / HDL\n",
|
||||
"||- s5 ltg, possibly log of serum triglycerides level\n",
|
||||
"||- s6 glu, blood sugar level|\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> 🎓 Remember, this is supervised learning, and we need a named 'y' target.\n",
|
||||
"\n",
|
||||
"Before you can manipulate data with R, you need to import the data into R's memory, or build a connection to the data that R can use to access the data remotely.\n",
|
||||
"\n",
|
||||
"> The [readr](https://readr.tidyverse.org/) package, which is part of the Tidyverse, provides a fast and friendly way to read rectangular data into R.\n",
|
||||
"\n",
|
||||
"Now, let's load the diabetes dataset provided in this source URL: <https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html>\n",
|
||||
"\n",
|
||||
"Also, we'll perform a sanity check on our data using `glimpse()` and display the first 5 rows using `slice()`.\n",
|
||||
"\n",
|
||||
"Before going any further, let's also introduce something you will encounter often in R code 🥁🥁: the pipe operator `%>%`\n",
|
||||
"\n",
|
||||
"The pipe operator (`%>%`) performs operations in logical sequence by passing an object forward into a function or call expression. You can think of the pipe operator as saying \"and then\" in your code."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "KM6iXLH996Cl"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Import the data set\r\n",
|
||||
"diabetes <- read_table2(file = \"https://www4.stat.ncsu.edu/~boos/var.select/diabetes.rwrite1.txt\")\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"# Get a glimpse and dimensions of the data\r\n",
|
||||
"glimpse(diabetes)\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"# Select the first 5 rows of the data\r\n",
|
||||
"diabetes %>% \r\n",
|
||||
" slice(1:5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "Z1geAMhM-bSP"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"`glimpse()` shows us that this data has 442 rows and 11 columns with all the columns being of data type `double` \n",
|
||||
"\n",
|
||||
"<br>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"> glimpse() and slice() are functions in [`dplyr`](https://dplyr.tidyverse.org/). Dplyr, part of the Tidyverse, is a grammar of data manipulation that provides a consistent set of verbs that help you solve the most common data manipulation challenges\n",
|
||||
"\n",
|
||||
"<br>\n",
|
||||
"\n",
|
||||
"Now that we have the data, let's narrow down to one feature (`bmi`) to target for this exercise. This will require us to select the desired columns. So, how do we do this?\n",
|
||||
"\n",
|
||||
"[`dplyr::select()`](https://dplyr.tidyverse.org/reference/select.html) allows us to *select* (and optionally rename) columns in a data frame."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "UwjVT1Hz-c3Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Select predictor feature `bmi` and outcome `y`\r\n",
|
||||
"diabetes_select <- diabetes %>% \r\n",
|
||||
" select(c(bmi, y))\r\n",
|
||||
"\r\n",
|
||||
"# Print the first 5 rows\r\n",
|
||||
"diabetes_select %>% \r\n",
|
||||
" slice(1:10)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "RDY1oAKI-m80"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 3. Training and Testing data\n",
|
||||
"\n",
|
||||
"It's common practice in supervised learning to *split* the data into two subsets; a (typically larger) set with which to train the model, and a smaller \"hold-back\" set with which to see how the model performed.\n",
|
||||
"\n",
|
||||
"Now that we have data ready, we can see if a machine can help determine a logical split between the numbers in this dataset. We can use the [rsample](https://tidymodels.github.io/rsample/) package, which is part of the Tidymodels framework, to create an object that contains the information on *how* to split the data, and then two more rsample functions to extract the created training and testing sets:\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "SDk668xK-tc3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"set.seed(2056)\r\n",
|
||||
"# Split 67% of the data for training and the rest for tesing\r\n",
|
||||
"diabetes_split <- diabetes_select %>% \r\n",
|
||||
" initial_split(prop = 0.67)\r\n",
|
||||
"\r\n",
|
||||
"# Extract the resulting train and test sets\r\n",
|
||||
"diabetes_train <- training(diabetes_split)\r\n",
|
||||
"diabetes_test <- testing(diabetes_split)\r\n",
|
||||
"\r\n",
|
||||
"# Print the first 3 rows of the training set\r\n",
|
||||
"diabetes_train %>% \r\n",
|
||||
" slice(1:10)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "EqtHx129-1h-"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 4. Train a linear regression model with Tidymodels\n",
|
||||
"\n",
|
||||
"Now we are ready to train our model!\n",
|
||||
"\n",
|
||||
"In Tidymodels, you specify models using `parsnip()` by specifying three concepts:\n",
|
||||
"\n",
|
||||
"- Model **type** differentiates models such as linear regression, logistic regression, decision tree models, and so forth.\n",
|
||||
"\n",
|
||||
"- Model **mode** includes common options like regression and classification; some model types support either of these while some only have one mode.\n",
|
||||
"\n",
|
||||
"- Model **engine** is the computational tool which will be used to fit the model. Often these are R packages, such as **`\"lm\"`** or **`\"ranger\"`**\n",
|
||||
"\n",
|
||||
"This modeling information is captured in a model specification, so let's build one!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "sBOS-XhB-6v7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Build a linear model specification\r\n",
|
||||
"lm_spec <- \r\n",
|
||||
" # Type\r\n",
|
||||
" linear_reg() %>% \r\n",
|
||||
" # Engine\r\n",
|
||||
" set_engine(\"lm\") %>% \r\n",
|
||||
" # Mode\r\n",
|
||||
" set_mode(\"regression\")\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"# Print the model specification\r\n",
|
||||
"lm_spec"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "20OwEw20--t3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"After a model has been *specified*, the model can be `estimated` or `trained` using the [`fit()`](https://parsnip.tidymodels.org/reference/fit.html) function, typically using a formula and some data.\n",
|
||||
"\n",
|
||||
"`y ~ .` means we'll fit `y` as the predicted quantity/target, explained by all the predictors/features ie, `.` (in this case, we only have one predictor: `bmi` )"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "_oDHs89k_CJj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Build a linear model specification\r\n",
|
||||
"lm_spec <- linear_reg() %>% \r\n",
|
||||
" set_engine(\"lm\") %>%\r\n",
|
||||
" set_mode(\"regression\")\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"# Train a linear regression model\r\n",
|
||||
"lm_mod <- lm_spec %>% \r\n",
|
||||
" fit(y ~ ., data = diabetes_train)\r\n",
|
||||
"\r\n",
|
||||
"# Print the model\r\n",
|
||||
"lm_mod"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "YlsHqd-q_GJQ"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"From the model output, we can see the coefficients learned during training. They represent the coefficients of the line of best fit that gives us the lowest overall error between the actual and predicted variable.\n",
|
||||
"<br>\n",
|
||||
"\n",
|
||||
"## 5. Make predictions on the test set\n",
|
||||
"\n",
|
||||
"Now that we've trained a model, we can use it to predict the disease progression y for the test dataset using [parsnip::predict()](https://parsnip.tidymodels.org/reference/predict.model_fit.html). This will be used to draw the line between data groups."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "kGZ22RQj_Olu"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Make predictions for the test set\r\n",
|
||||
"predictions <- lm_mod %>% \r\n",
|
||||
" predict(new_data = diabetes_test)\r\n",
|
||||
"\r\n",
|
||||
"# Print out some of the predictions\r\n",
|
||||
"predictions %>% \r\n",
|
||||
" slice(1:5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "nXHbY7M2_aao"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Woohoo! 💃🕺 We just trained a model and used it to make predictions!\n",
|
||||
"\n",
|
||||
"When making predictions, the tidymodels convention is to always produce a tibble/data frame of results with standardized column names. This makes it easy to combine the original data and the predictions in a usable format for subsequent operations such as plotting.\n",
|
||||
"\n",
|
||||
"`dplyr::bind_cols()` efficiently binds multiple data frames column."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "R_JstwUY_bIs"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Combine the predictions and the original test set\r\n",
|
||||
"results <- diabetes_test %>% \r\n",
|
||||
" bind_cols(predictions)\r\n",
|
||||
"\r\n",
|
||||
"\r\n",
|
||||
"results %>% \r\n",
|
||||
" slice(1:5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "RybsMJR7_iI8"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 6. Plot modelling results\n",
|
||||
"\n",
|
||||
"Now, its time to see this visually 📈. We'll create a scatter plot of all the `y` and `bmi` values of the test set, then use the predictions to draw a line in the most appropriate place, between the model's data groupings.\n",
|
||||
"\n",
|
||||
"R has several systems for making graphs, but `ggplot2` is one of the most elegant and most versatile. This allows you to compose graphs by **combining independent components**."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "XJbYbMZW_n_s"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Set a theme for the plot\r\n",
|
||||
"theme_set(theme_light())\r\n",
|
||||
"# Create a scatter plot\r\n",
|
||||
"results %>% \r\n",
|
||||
" ggplot(aes(x = bmi)) +\r\n",
|
||||
" # Add a scatter plot\r\n",
|
||||
" geom_point(aes(y = y), size = 1.6) +\r\n",
|
||||
" # Add a line plot\r\n",
|
||||
" geom_line(aes(y = .pred), color = \"blue\", size = 1.5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "R9tYp3VW_sTn"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"> ✅ Think a bit about what's going on here. A straight line is running through many small dots of data, but what is it doing exactly? Can you see how you should be able to use this line to predict where a new, unseen data point should fit in relationship to the plot's y axis? Try to put into words the practical use of this model.\n",
|
||||
"\n",
|
||||
"Congratulations, you built your first linear regression model, created a prediction with it, and displayed it in a plot!\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "zrPtHIxx_tNI"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
---
|
||||
title: 'Build a regression model: Get started with R and Tidymodels for regression models'
|
||||
output:
|
||||
html_document:
|
||||
df_print: paged
|
||||
theme: flatly
|
||||
highlight: breezedark
|
||||
toc: yes
|
||||
toc_float: yes
|
||||
code_download: yes
|
||||
---
|
||||
|
||||
## Introduction to Regression - Lesson 1
|
||||
|
||||
#### Putting it into perspective
|
||||
|
||||
✅ There are many types of regression methods, and which one you pick depends on the answer you're looking for. If you want to predict the probable height for a person of a given age, you'd use `linear regression`, as you're seeking a **numeric value**. If you're interested in discovering whether a type of cuisine should be considered vegan or not, you're looking for a **category assignment** so you would use `logistic regression`. You'll learn more about logistic regression later. Think a bit about some questions you can ask of data, and which of these methods would be more appropriate.
|
||||
|
||||
In this section, you will work with a [small dataset about diabetes](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html). Imagine that you wanted to test a treatment for diabetic patients. Machine Learning models might help you determine which patients would respond better to the treatment, based on combinations of variables. Even a very basic regression model, when visualized, might show information about variables that would help you organize your theoretical clinical trials.
|
||||
|
||||
That said, let's get started on this task!
|
||||
|
||||
{width="630"}
|
||||
|
||||
## 1. Loading up our tool set
|
||||
|
||||
For this task, we'll require the following packages:
|
||||
|
||||
- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!
|
||||
|
||||
- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.
|
||||
|
||||
You can have them installed as:
|
||||
|
||||
`install.packages(c("tidyverse", "tidymodels"))`
|
||||
|
||||
The script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.
|
||||
|
||||
```{r, message=F, warning=F}
|
||||
if (!require("pacman")) install.packages("pacman")
|
||||
pacman::p_load(tidyverse, tidymodels)
|
||||
```
|
||||
|
||||
Now, let's load these awesome packages and make them available in our current R session. (This is for mere illustration, `pacman::p_load()` already did that for you)
|
||||
|
||||
```{r load_tidy_verse_models, message=F, warning=F}
|
||||
# load the core Tidyverse packages
|
||||
library(tidyverse)
|
||||
|
||||
# load the core Tidymodels packages
|
||||
library(tidymodels)
|
||||
|
||||
|
||||
```
|
||||
|
||||
## 2. The diabetes dataset
|
||||
|
||||
In this exercise, we'll put our regression skills into display by making predictions on a diabetes dataset. The [diabetes dataset](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.rwrite1.txt) includes `442 samples` of data around diabetes, with 10 predictor feature variables, `age`, `sex`, `body mass index`, `average blood pressure`, and `six blood serum measurements` as well as an outcome variable `y`: a quantitative measure of disease progression one year after baseline.
|
||||
|
||||
+----------------------------+------------------------------------------------------------------------------------+
|
||||
| **Number of observations** | **442** |
|
||||
+============================+====================================================================================+
|
||||
| **Number of predictors** | First 10 columns are numeric predictive values |
|
||||
+----------------------------+------------------------------------------------------------------------------------+
|
||||
| **Outcome/Target** | Column 11 is a quantitative measure of disease progression one year after baseline |
|
||||
+----------------------------+------------------------------------------------------------------------------------+
|
||||
| **Predictor Information** | - age age in years |
|
||||
| | - sex |
|
||||
| | - bmi body mass index |
|
||||
| | - bp average blood pressure |
|
||||
| | - s1 tc, total serum cholesterol |
|
||||
| | - s2 ldl, low-density lipoproteins |
|
||||
| | - s3 hdl, high-density lipoproteins |
|
||||
| | - s4 tch, total cholesterol / HDL |
|
||||
| | - s5 ltg, possibly log of serum triglycerides level |
|
||||
| | - s6 glu, blood sugar level |
|
||||
+----------------------------+------------------------------------------------------------------------------------+
|
||||
|
||||
> 🎓 Remember, this is supervised learning, and we need a named 'y' target.
|
||||
|
||||
Before you can manipulate data with R, you need to import the data into R's memory, or build a connection to the data that R can use to access the data remotely.\
|
||||
|
||||
> The [readr](https://readr.tidyverse.org/) package, which is part of the Tidyverse, provides a fast and friendly way to read rectangular data into R.
|
||||
|
||||
Now, let's load the diabetes dataset provided in this source URL: <https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html>
|
||||
|
||||
Also, we'll perform a sanity check on our data using `glimpse()` and dsiplay the first 5 rows using `slice()`.
|
||||
|
||||
Before going any further, let's introduce something you will encounter quite often in R code: the pipe operator `%>%`
|
||||
|
||||
The pipe operator (`%>%`) performs operations in logical sequence by passing an object forward into a function or call expression. You can think of the pipe operator as saying "and then" in your code.\
|
||||
|
||||
```{r load_dataset, message=F, warning=F}
|
||||
# Import the data set
|
||||
diabetes <- read_table2(file = "https://www4.stat.ncsu.edu/~boos/var.select/diabetes.rwrite1.txt")
|
||||
|
||||
|
||||
# Get a glimpse and dimensions of the data
|
||||
glimpse(diabetes)
|
||||
|
||||
|
||||
# Select the first 5 rows of the data
|
||||
diabetes %>%
|
||||
slice(1:5)
|
||||
|
||||
```
|
||||
|
||||
`glimpse()` shows us that this data has 442 rows and 11 columns with all the columns being of data type `double`
|
||||
|
||||
> glimpse() and slice() are functions in [`dplyr`](https://dplyr.tidyverse.org/). Dplyr, part of the Tidyverse, is a grammar of data manipulation that provides a consistent set of verbs that help you solve the most common data manipulation challenges
|
||||
|
||||
Now that we have the data, let's narrow down to one feature (`bmi`) to target for this exercise. This will require us to select the desired columns. So, how do we do this?
|
||||
|
||||
[`dplyr::select()`](https://dplyr.tidyverse.org/reference/select.html) allows us to *select* (and optionally rename) columns in a data frame.
|
||||
|
||||
```{r select, message=F, warning=F}
|
||||
# Select predictor feature `bmi` and outcome `y`
|
||||
diabetes_select <- diabetes %>%
|
||||
select(c(bmi, y))
|
||||
|
||||
# Print the first 5 rows
|
||||
diabetes_select %>%
|
||||
slice(1:5)
|
||||
```
|
||||
|
||||
## 3. Training and Testing data
|
||||
|
||||
It's common practice in supervised learning to *split* the data into two subsets; a (typically larger) set with which to train the model, and a smaller "hold-back" set with which to see how the model performed.
|
||||
|
||||
Now that we have data ready, we can see if a machine can help determine a logical split between the numbers in this dataset. We can use the [rsample](https://tidymodels.github.io/rsample/) package, which is part of the Tidymodels framework, to create an object that contains the information on *how* to split the data, and then two more rsample functions to extract the created training and testing sets:
|
||||
|
||||
```{r split, message=F, warning=F}
|
||||
set.seed(2056)
|
||||
# Split 67% of the data for training and the rest for testing
|
||||
diabetes_split <- diabetes_select %>%
|
||||
initial_split(prop = 0.67)
|
||||
|
||||
# Extract the resulting train and test sets
|
||||
diabetes_train <- training(diabetes_split)
|
||||
diabetes_test <- testing(diabetes_split)
|
||||
|
||||
# Print the first 3 rows of the training set
|
||||
diabetes_train %>%
|
||||
slice(1:3)
|
||||
|
||||
```
|
||||
|
||||
## 4. Train a linear regression model with Tidymodels
|
||||
|
||||
Now we are ready to train our model!
|
||||
|
||||
In Tidymodels, you specify models using `parsnip()` by specifying three concepts:
|
||||
|
||||
- Model **type** differentiates models such as linear regression, logistic regression, decision tree models, and so forth.
|
||||
|
||||
- Model **mode** includes common options like regression and classification; some model types support either of these while some only have one mode.
|
||||
|
||||
- Model **engine** is the computational tool which will be used to fit the model. Often these are R packages, such as **`"lm"`** or **`"ranger"`**
|
||||
|
||||
This modeling information is captured in a model specification, so let's build one!
|
||||
|
||||
```{r lm_model_spec, message=F, warning=F}
|
||||
# Build a linear model specification
|
||||
lm_spec <-
|
||||
# Type
|
||||
linear_reg() %>%
|
||||
# Engine
|
||||
set_engine("lm") %>%
|
||||
# Mode
|
||||
set_mode("regression")
|
||||
|
||||
|
||||
# Print the model specification
|
||||
lm_spec
|
||||
|
||||
```
|
||||
|
||||
After a model has been *specified*, the model can be `estimated` or `trained` using the [`fit()`](https://parsnip.tidymodels.org/reference/fit.html) function, typically using a formula and some data.
|
||||
|
||||
`y ~ .` means we'll fit `y` as the predicted quantity/target, explained by all the predictors/features ie, `.` (in this case, we only have one predictor: `bmi` )
|
||||
|
||||
```{r train, message=F, warning=F}
|
||||
# Build a linear model specification
|
||||
lm_spec <- linear_reg() %>%
|
||||
set_engine("lm") %>%
|
||||
set_mode("regression")
|
||||
|
||||
|
||||
# Train a linear regression model
|
||||
lm_mod <- lm_spec %>%
|
||||
fit(y ~ ., data = diabetes_train)
|
||||
|
||||
# Print the model
|
||||
lm_mod
|
||||
```
|
||||
|
||||
From the model output, we can see the coefficients learned during training. They represent the coefficients of the line of best fit that gives us the lowest overall error between the actual and predicted variable.
|
||||
|
||||
## 5. Make predictions on the test set
|
||||
|
||||
Now that we've trained a model, we can use it to predict the disease progression y for the test dataset using [parsnip::predict()](https://parsnip.tidymodels.org/reference/predict.model_fit.html). This will be used to draw the line between data groups.
|
||||
|
||||
```{r test, message=F, warning=F}
|
||||
# Make predictions for the test set
|
||||
predictions <- lm_mod %>%
|
||||
predict(new_data = diabetes_test)
|
||||
|
||||
# Print out some of the predictions
|
||||
predictions %>%
|
||||
slice(1:5)
|
||||
```
|
||||
|
||||
Woohoo! 💃🕺 We just trained a model and used it to make predictions!
|
||||
|
||||
When making predictions, the tidymodels convention is to always produce a tibble/data frame of results with standardized column names. This makes it easy to combine the original data and the predictions in a usable format for subsequent operations such as plotting.
|
||||
|
||||
`dplyr::bind_cols()` efficiently binds multiple data frames column.
|
||||
|
||||
```{r test_pred, message=F, warning=F}
|
||||
# Combine the predictions and the original test set
|
||||
results <- diabetes_test %>%
|
||||
bind_cols(predictions)
|
||||
|
||||
|
||||
results %>%
|
||||
slice(1:5)
|
||||
```
|
||||
|
||||
## 6. Plot modelling results
|
||||
|
||||
Now, its time to see this visually 📈. We'll create a scatter plot of all the `y` and `bmi` values of the test set, then use the predictions to draw a line in the most appropriate place, between the model's data groupings.
|
||||
|
||||
R has several systems for making graphs, but `ggplot2` is one of the most elegant and most versatile. This allows you to compose graphs by **combining independent components**.
|
||||
|
||||
```{r plot_pred, message=F, warning=F}
|
||||
# Set a theme for the plot
|
||||
theme_set(theme_light())
|
||||
# Create a scatter plot
|
||||
results %>%
|
||||
ggplot(aes(x = bmi)) +
|
||||
# Add a scatter plot
|
||||
geom_point(aes(y = y), size = 1.6) +
|
||||
# Add a line plot
|
||||
geom_line(aes(y = .pred), color = "blue", size = 1.5)
|
||||
|
||||
```
|
||||
|
||||
> ✅ Think a bit about what's going on here. A straight line is running through many small dots of data, but what is it doing exactly? Can you see how you should be able to use this line to predict where a new, unseen data point should fit in relationship to the plot's y axis? Try to put into words the practical use of this model.
|
||||
|
||||
Congratulations, you built your first linear regression model, created a prediction with it, and displayed it in a plot!
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,286 @@
|
||||
# Build a regression model using Scikit-learn: prepare and visualize data
|
||||
|
||||
|
||||
|
||||
Infographic by [Dasani Madipalli](https://twitter.com/dasani_decoded)
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
> ### [This lesson is available in R!](./solution/R/lesson_2.html)
|
||||
|
||||
## Introduction
|
||||
|
||||
Now that you are set up with the tools you need to start tackling machine learning model building with Scikit-learn, you are ready to start asking questions of your data. As you work with data and apply ML solutions, it's very important to understand how to ask the right question to properly unlock the potentials of your dataset.
|
||||
|
||||
In this lesson, you will learn:
|
||||
|
||||
- How to prepare your data for model-building.
|
||||
- How to use Matplotlib for data visualization.
|
||||
- How to use Seaborn for more expressive data visualization.
|
||||
|
||||
## Asking the right question of your data
|
||||
|
||||
The question you need answered will determine what type of ML algorithms you will leverage. And the quality of the answer you get back will be heavily dependent on the nature of your data.
|
||||
|
||||
Take a look at the [data](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/data/US-pumpkins.csv) provided for this lesson. You can open this .csv file in VS Code. A quick skim immediately shows that there are blanks and a mix of strings and numeric data. There's also a strange column called 'Package' where the data is a mix between 'sacks', 'bins' and other values. The data, in fact, is a bit of a mess.
|
||||
|
||||
[](https://youtu.be/5qGjczWTrDQ "ML for beginners - How to Analyze and Clean a Dataset")
|
||||
|
||||
> 🎥 Click the image above for a short video working through preparing the data for this lesson.
|
||||
|
||||
In fact, it is not very common to be gifted a dataset that is completely ready to use to create a ML model out of the box. In this lesson, you will learn how to prepare a raw dataset using standard Python libraries. You will also learn various techniques to visualize the data.
|
||||
|
||||
## Case study: 'the pumpkin market'
|
||||
|
||||
In this folder you will find a .csv file in the root `data` folder called [US-pumpkins.csv](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/data/US-pumpkins.csv) which includes 1757 lines of data about the market for pumpkins, sorted into groupings by city. This is raw data extracted from the [Specialty Crops Terminal Markets Standard Reports](https://www.marketnews.usda.gov/mnp/fv-report-config-step1?type=termPrice) distributed by the United States Department of Agriculture.
|
||||
|
||||
### Preparing data
|
||||
|
||||
This data is in the public domain. It can be downloaded in many separate files, per city, from the USDA web site. To avoid too many separate files, we have concatenated all the city data into one spreadsheet, thus we have already _prepared_ the data a bit. Next, let's take a closer look at the data.
|
||||
|
||||
### The pumpkin data - early conclusions
|
||||
|
||||
What do you notice about this data? You already saw that there is a mix of strings, numbers, blanks and strange values that you need to make sense of.
|
||||
|
||||
What question can you ask of this data, using a Regression technique? What about "Predict the price of a pumpkin for sale during a given month". Looking again at the data, there are some changes you need to make to create the data structure necessary for the task.
|
||||
## Exercise - analyze the pumpkin data
|
||||
|
||||
Let's use [Pandas](https://pandas.pydata.org/), (the name stands for `Python Data Analysis`) a tool very useful for shaping data, to analyze and prepare this pumpkin data.
|
||||
|
||||
### First, check for missing dates
|
||||
|
||||
You will first need to take steps to check for missing dates:
|
||||
|
||||
1. Convert the dates to a month format (these are US dates, so the format is `MM/DD/YYYY`).
|
||||
2. Extract the month to a new column.
|
||||
|
||||
Open the _notebook.ipynb_ file in Visual Studio Code and import the spreadsheet in to a new Pandas dataframe.
|
||||
|
||||
1. Use the `head()` function to view the first five rows.
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
pumpkins = pd.read_csv('../data/US-pumpkins.csv')
|
||||
pumpkins.head()
|
||||
```
|
||||
|
||||
✅ What function would you use to view the last five rows?
|
||||
|
||||
1. Check if there is missing data in the current dataframe:
|
||||
|
||||
```python
|
||||
pumpkins.isnull().sum()
|
||||
```
|
||||
|
||||
There is missing data, but maybe it won't matter for the task at hand.
|
||||
|
||||
1. To make your dataframe easier to work with, select only the columns you need, using the `loc` function which extracts from the original dataframe a group of rows (passed as first parameter) and columns (passed as second parameter). The expression `:` in the case below means "all rows".
|
||||
|
||||
```python
|
||||
columns_to_select = ['Package', 'Low Price', 'High Price', 'Date']
|
||||
pumpkins = pumpkins.loc[:, columns_to_select]
|
||||
```
|
||||
|
||||
### Second, determine average price of pumpkin
|
||||
|
||||
Think about how to determine the average price of a pumpkin in a given month. What columns would you pick for this task? Hint: you'll need 3 columns.
|
||||
|
||||
Solution: take the average of the `Low Price` and `High Price` columns to populate the new Price column, and convert the Date column to only show the month. Fortunately, according to the check above, there is no missing data for dates or prices.
|
||||
|
||||
1. To calculate the average, add the following code:
|
||||
|
||||
```python
|
||||
price = (pumpkins['Low Price'] + pumpkins['High Price']) / 2
|
||||
|
||||
month = pd.DatetimeIndex(pumpkins['Date']).month
|
||||
|
||||
```
|
||||
|
||||
✅ Feel free to print any data you'd like to check using `print(month)`.
|
||||
|
||||
2. Now, copy your converted data into a fresh Pandas dataframe:
|
||||
|
||||
```python
|
||||
new_pumpkins = pd.DataFrame({'Month': month, 'Package': pumpkins['Package'], 'Low Price': pumpkins['Low Price'],'High Price': pumpkins['High Price'], 'Price': price})
|
||||
```
|
||||
|
||||
Printing out your dataframe will show you a clean, tidy dataset on which you can build your new regression model.
|
||||
|
||||
### But wait! There's something odd here
|
||||
|
||||
If you look at the `Package` column, pumpkins are sold in many different configurations. Some are sold in '1 1/9 bushel' measures, and some in '1/2 bushel' measures, some per pumpkin, some per pound, and some in big boxes with varying widths.
|
||||
|
||||
> Pumpkins seem very hard to weigh consistently
|
||||
|
||||
Digging into the original data, it's interesting that anything with `Unit of Sale` equalling 'EACH' or 'PER BIN' also have the `Package` type per inch, per bin, or 'each'. Pumpkins seem to be very hard to weigh consistently, so let's filter them by selecting only pumpkins with the string 'bushel' in their `Package` column.
|
||||
|
||||
1. Add a filter at the top of the file, under the initial .csv import:
|
||||
|
||||
```python
|
||||
pumpkins = pumpkins[pumpkins['Package'].str.contains('bushel', case=True, regex=True)]
|
||||
```
|
||||
|
||||
If you print the data now, you can see that you are only getting the 415 or so rows of data containing pumpkins by the bushel.
|
||||
|
||||
### But wait! There's one more thing to do
|
||||
|
||||
Did you notice that the bushel amount varies per row? You need to normalize the pricing so that you show the pricing per bushel, so do some math to standardize it.
|
||||
|
||||
1. Add these lines after the block creating the new_pumpkins dataframe:
|
||||
|
||||
```python
|
||||
new_pumpkins.loc[new_pumpkins['Package'].str.contains('1 1/9'), 'Price'] = price/(1 + 1/9)
|
||||
|
||||
new_pumpkins.loc[new_pumpkins['Package'].str.contains('1/2'), 'Price'] = price/(1/2)
|
||||
```
|
||||
|
||||
✅ According to [The Spruce Eats](https://www.thespruceeats.com/how-much-is-a-bushel-1389308), a bushel's weight depends on the type of produce, as it's a volume measurement. "A bushel of tomatoes, for example, is supposed to weigh 56 pounds... Leaves and greens take up more space with less weight, so a bushel of spinach is only 20 pounds." It's all pretty complicated! Let's not bother with making a bushel-to-pound conversion, and instead price by the bushel. All this study of bushels of pumpkins, however, goes to show how very important it is to understand the nature of your data!
|
||||
|
||||
Now, you can analyze the pricing per unit based on their bushel measurement. If you print out the data one more time, you can see how it's standardized.
|
||||
|
||||
✅ Did you notice that pumpkins sold by the half-bushel are very expensive? Can you figure out why? Hint: little pumpkins are way pricier than big ones, probably because there are so many more of them per bushel, given the unused space taken by one big hollow pie pumpkin.
|
||||
|
||||
## Visualization Strategies
|
||||
|
||||
Part of the data scientist's role is to demonstrate the quality and nature of the data they are working with. To do this, they often create interesting visualizations, or plots, graphs, and charts, showing different aspects of data. In this way, they are able to visually show relationships and gaps that are otherwise hard to uncover.
|
||||
|
||||
[](https://youtu.be/SbUkxH6IJo0 "ML for beginners - How to Visualize Data with Matplotlib")
|
||||
|
||||
> 🎥 Click the image above for a short video working through visualizing the data for this lesson.
|
||||
|
||||
Visualizations can also help determine the machine learning technique most appropriate for the data. A scatterplot that seems to follow a line, for example, indicates that the data is a good candidate for a linear regression exercise.
|
||||
|
||||
One data visualization library that works well in Jupyter notebooks is [Matplotlib](https://matplotlib.org/) (which you also saw in the previous lesson).
|
||||
|
||||
> Get more experience with data visualization in [these tutorials](https://docs.microsoft.com/learn/modules/explore-analyze-data-with-python?WT.mc_id=academic-77952-leestott).
|
||||
|
||||
## Exercise - experiment with Matplotlib
|
||||
|
||||
Try to create some basic plots to display the new dataframe you just created. What would a basic line plot show?
|
||||
|
||||
1. Import Matplotlib at the top of the file, under the Pandas import:
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
```
|
||||
|
||||
1. Rerun the entire notebook to refresh.
|
||||
1. At the bottom of the notebook, add a cell to plot the data as a box:
|
||||
|
||||
```python
|
||||
price = new_pumpkins.Price
|
||||
month = new_pumpkins.Month
|
||||
plt.scatter(price, month)
|
||||
plt.show()
|
||||
```
|
||||
|
||||
|
||||
|
||||
Is this a useful plot? Does anything about it surprise you?
|
||||
|
||||
It's not particularly useful as all it does is display in your data as a spread of points in a given month.
|
||||
|
||||
### Make it useful
|
||||
|
||||
To get charts to display useful data, you usually need to group the data somehow. Let's try creating a plot where the y axis shows the months and the data demonstrates the distribution of data.
|
||||
|
||||
1. Add a cell to create a grouped bar chart:
|
||||
|
||||
```python
|
||||
new_pumpkins.groupby(['Month'])['Price'].mean().plot(kind='bar')
|
||||
plt.ylabel("Pumpkin Price")
|
||||
```
|
||||
|
||||
|
||||
|
||||
This is a more useful data visualization! It seems to indicate that the highest price for pumpkins occurs in September and October. Does that meet your expectation? Why or why not?
|
||||
|
||||
## Exercise - experiment with Seaborn
|
||||
|
||||
Matplotlib is powerful, but it can take a lot of code to produce a polished chart. [Seaborn](https://seaborn.pydata.org/) is a library built _on top of_ Matplotlib that is designed for statistical data visualization. It works directly with Pandas dataframes, applies attractive default styles, and lets you create informative plots with far less code. Because Seaborn returns Matplotlib objects, you can still use everything you already know about Matplotlib to fine-tune the result.
|
||||
|
||||
> If you don't already have Seaborn installed, install it with `pip install seaborn`.
|
||||
|
||||
1. Import Seaborn at the top of the notebook, under the other imports. It is conventionally imported as `sns`:
|
||||
|
||||
```python
|
||||
import seaborn as sns
|
||||
```
|
||||
|
||||
### Scatter plots to show relationships
|
||||
|
||||
A big part of exploring data before building a model is looking for _relationships_ between variables. A [scatter plot](https://en.wikipedia.org/wiki/Scatter_plot) is one of the best tools for this: if the points seem to follow a line, the two variables may be correlated, which is a good sign that a linear regression model could work.
|
||||
|
||||
1. Recreate the price-to-month scatter plot from before, this time using Seaborn's [`relplot()`](https://seaborn.pydata.org/generated/seaborn.relplot.html) (relational plot), which works directly with your dataframe columns:
|
||||
|
||||
```python
|
||||
sns.relplot(x="Price", y="Month", data=new_pumpkins)
|
||||
```
|
||||
|
||||
|
||||
|
||||
Notice how you pass the _column names_ and the dataframe, and Seaborn takes care of the axis labels for you.
|
||||
|
||||
2. You can switch to a line plot by passing `kind="line"`. Seaborn even draws a shaded band showing the confidence interval around the line:
|
||||
|
||||
```python
|
||||
sns.relplot(x="Price", y="Month", kind="line", data=new_pumpkins)
|
||||
```
|
||||
|
||||
|
||||
|
||||
This particular data is quite noisy, so a line plot isn't the clearest choice here — but it shows how easily you can change chart types in Seaborn.
|
||||
|
||||
### Bar charts to show distributions
|
||||
|
||||
Earlier you grouped the data by hand to create a bar chart with Matplotlib. Seaborn's [`catplot()`](https://seaborn.pydata.org/generated/seaborn.catplot.html) (categorical plot) can do the grouping and aggregation for you. By default `kind="bar"` shows the mean of each category along with a black line indicating the confidence interval.
|
||||
|
||||
1. Create a bar chart of average price per month:
|
||||
|
||||
```python
|
||||
sns.catplot(x="Month", y="Price", data=new_pumpkins, kind="bar")
|
||||
```
|
||||
|
||||
|
||||
|
||||
This confirms what you saw with Matplotlib — prices peak around September and October — but Seaborn also visualizes how much the price _varies_ within each month.
|
||||
|
||||
### Heatmaps to show correlations
|
||||
|
||||
Scatter plots compare two variables at a time. When you have several numeric columns, a [heatmap](https://en.wikipedia.org/wiki/Heat_map) lets you view the strength of the relationship between _every_ pair of columns at once. This is a common way to spot which features are most correlated before choosing what to feed into a model (and the same kind of chart is later used to display confusion matrices in classification).
|
||||
|
||||
1. Build a correlation matrix with Pandas, then draw it with Seaborn's [`heatmap()`](https://seaborn.pydata.org/generated/seaborn.heatmap.html). The `annot=True` option prints the correlation values on each cell:
|
||||
|
||||
```python
|
||||
correlations = new_pumpkins[['Month', 'Low Price', 'High Price', 'Price']].corr()
|
||||
sns.heatmap(correlations, annot=True, cmap="coolwarm")
|
||||
```
|
||||
|
||||
|
||||
|
||||
Values close to `1` (or `-1`) mean the columns are strongly _linearly_ correlated. Notice how `Low Price` and `High Price` are almost perfectly correlated. `Month`, on the other hand, shows only a weak linear correlation with price — even though the bar chart above revealed a clear seasonal peak in September and October. That's an important lesson: the correlation coefficient only measures _straight-line_ relationships, so it can miss seasonal or otherwise non-linear patterns. ✅ Why is it useful to look at both a heatmap *and* charts like the bar chart before deciding which columns to use?
|
||||
|
||||
### Matplotlib or Seaborn?
|
||||
|
||||
Both libraries are worth knowing:
|
||||
|
||||
- **Matplotlib** gives you fine-grained control over every element of a chart and is the foundation almost every other Python plotting library builds on.
|
||||
- **Seaborn** provides higher-level functions and attractive defaults for statistical charts, works directly with dataframes, and is often quicker for exploratory data analysis.
|
||||
|
||||
A common workflow is to reach for Seaborn to explore your data quickly, then drop down to Matplotlib when you need to customize the details.
|
||||
|
||||
---
|
||||
|
||||
## 🚀Challenge
|
||||
|
||||
Explore the different types of visualization that Matplotlib and Seaborn offer. Which types are most appropriate for regression problems?
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
Take a look at the many ways to visualize data. Make a list of the various libraries available and note which are best for given types of tasks, for example 2D visualizations vs. 3D visualizations. What do you discover?
|
||||
|
||||
## Assignment
|
||||
|
||||
[Exploring visualization](assignment.md)
|
||||
@@ -0,0 +1,8 @@
|
||||
# Exploring Visualizations
|
||||
|
||||
There are several different libraries that are available for data visualization. Create some visualizations using the Pumpkin data in this lesson with matplotlib and seaborn in a sample notebook. Which libraries are easier to work with?
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | --------- | -------- | ----------------- |
|
||||
| | A notebook is submitted with two explorations/visualizations | A notebook is submitted with one explorations/visualizations | A notebook is not submitted |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 213 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.0 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 172 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.3-final"
|
||||
},
|
||||
"orig_nbformat": 2,
|
||||
"kernelspec": {
|
||||
"name": "python3",
|
||||
"display_name": "Python 3",
|
||||
"language": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2,
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
This is a temporary placeholder
|
||||
@@ -0,0 +1,664 @@
|
||||
{
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2,
|
||||
"metadata": {
|
||||
"colab": {
|
||||
"name": "lesson_2-R.ipynb",
|
||||
"provenance": [],
|
||||
"collapsed_sections": [],
|
||||
"toc_visible": true
|
||||
},
|
||||
"kernelspec": {
|
||||
"name": "ir",
|
||||
"display_name": "R"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "R"
|
||||
}
|
||||
},
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"# Build a regression model: prepare and visualize data\n",
|
||||
"\n",
|
||||
"## **Linear Regression for Pumpkins - Lesson 2**\n",
|
||||
"#### Introduction\n",
|
||||
"\n",
|
||||
"Now that you are set up with the tools you need to start tackling machine learning model building with Tidymodels and the Tidyverse, you are ready to start asking questions of your data. As you work with data and apply ML solutions, it's very important to understand how to ask the right question to properly unlock the potentials of your dataset.\n",
|
||||
"\n",
|
||||
"In this lesson, you will learn:\n",
|
||||
"\n",
|
||||
"- How to prepare your data for model-building.\n",
|
||||
"\n",
|
||||
"- How to use `ggplot2` for data visualization.\n",
|
||||
"\n",
|
||||
"The question you need answered will determine what type of ML algorithms you will leverage. And the quality of the answer you get back will be heavily dependent on the nature of your data.\n",
|
||||
"\n",
|
||||
"Let's see this by working through a practical exercise.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/unruly_data.jpg\"\n",
|
||||
" width=\"700\"/>\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<!--<br>Artwork by \\@allison_horst-->"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Pg5aexcOPqAZ"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 1. Importing pumpkins data and summoning the Tidyverse\n",
|
||||
"\n",
|
||||
"We'll require the following packages to slice and dice this lesson:\n",
|
||||
"\n",
|
||||
"- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!\n",
|
||||
"\n",
|
||||
"You can have them installed as:\n",
|
||||
"\n",
|
||||
"`install.packages(c(\"tidyverse\"))`\n",
|
||||
"\n",
|
||||
"The script below checks whether you have the packages required to complete this module and installs them for you in case some are missing."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "dc5WhyVdXAjR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"suppressWarnings(if(!require(\"pacman\")) install.packages(\"pacman\"))\n",
|
||||
"pacman::p_load(tidyverse)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "GqPYUZgfXOBt"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now, let's fire up some packages and load the [data](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/data/US-pumpkins.csv) provided for this lesson!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "kvjDTPDSXRr2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Load the core Tidyverse packages\n",
|
||||
"library(tidyverse)\n",
|
||||
"\n",
|
||||
"# Import the pumpkins data\n",
|
||||
"pumpkins <- read_csv(file = \"https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/2-Regression/data/US-pumpkins.csv\")\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Get a glimpse and dimensions of the data\n",
|
||||
"glimpse(pumpkins)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Print the first 50 rows of the data set\n",
|
||||
"pumpkins %>% \n",
|
||||
" slice_head(n =50)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "VMri-t2zXqgD"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"A quick `glimpse()` immediately shows that there are blanks and a mix of strings (`chr`) and numeric data (`dbl`). The `Date` is of type character and there's also a strange column called `Package` where the data is a mix between `sacks`, `bins` and other values. The data, in fact, is a bit of a mess 😤.\n",
|
||||
"\n",
|
||||
"In fact, it is not very common to be gifted a dataset that is completely ready to use to create a ML model out of the box. But worry not, in this lesson, you will learn how to prepare a raw dataset using standard R libraries 🧑🔧. You will also learn various techniques to visualize the data.📈📊\n",
|
||||
"<br>\n",
|
||||
"\n",
|
||||
"> A refresher: The pipe operator (`%>%`) performs operations in logical sequence by passing an object forward into a function or call expression. You can think of the pipe operator as saying \"and then\" in your code.\n",
|
||||
"\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "REWcIv9yX29v"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 2. Check for missing data\n",
|
||||
"\n",
|
||||
"One of the most common issues data scientists need to deal with is incomplete or missing data. R represents missing, or unknown values, with special sentinel value: `NA` (Not Available).\n",
|
||||
"\n",
|
||||
"So how would we know that the data frame contains missing values?\n",
|
||||
"<br>\n",
|
||||
"- One straight forward way would be to use the base R function `anyNA` which returns the logical objects `TRUE` or `FALSE`"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Zxfb3AM5YbUe"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"pumpkins %>% \n",
|
||||
" anyNA()"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "G--DQutAYltj"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Great, there seems to be some missing data! That's a good place to start.\n",
|
||||
"\n",
|
||||
"- Another way would be to use the function `is.na()` that indicates which individual column elements are missing with a logical `TRUE`."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mU-7-SB6YokF"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"pumpkins %>% \n",
|
||||
" is.na() %>% \n",
|
||||
" head(n = 7)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "W-DxDOR4YxSW"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Okay, got the job done but with a large data frame such as this, it would be inefficient and practically impossible to review all of the rows and columns individually😴.\n",
|
||||
"\n",
|
||||
"- A more intuitive way would be to calculate the sum of the missing values for each column:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "xUWxipKYY0o7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"pumpkins %>% \n",
|
||||
" is.na() %>% \n",
|
||||
" colSums()"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "ZRBWV6P9ZArL"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Much better! There is missing data, but maybe it won't matter for the task at hand. Let's see what further analysis brings forth.\n",
|
||||
"\n",
|
||||
"> Along with the awesome sets of packages and functions, R has a very good documentation. For instance, use `help(colSums)` or `?colSums` to find out more about the function."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "9gv-crB6ZD1Y"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 3. Dplyr: A Grammar of Data Manipulation\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/dplyr_wrangling.png\"\n",
|
||||
" width=\"569\"/>\n",
|
||||
" <figcaption>Artwork by @allison_horst</figcaption>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<!--<br/>Artwork by \\@allison_horst-->"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "o4jLY5-VZO2C"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"[`dplyr`](https://dplyr.tidyverse.org/), a package in the Tidyverse, is a grammar of data manipulation that provides a consistent set of verbs that help you solve the most common data manipulation challenges. In this section, we'll explore some of dplyr's verbs!\n",
|
||||
"<br>\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "i5o33MQBZWWw"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"#### dplyr::select()\n",
|
||||
"\n",
|
||||
"`select()` is a function in the package `dplyr` which helps you pick columns to keep or exclude.\n",
|
||||
"\n",
|
||||
"To make your data frame easier to work with, drop several of its columns, using `select()`, keeping only the columns you need.\n",
|
||||
"\n",
|
||||
"For instance, in this exercise, our analysis will involve the columns `Package`, `Low Price`, `High Price` and `Date`. Let's select these columns."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "x3VGMAGBZiUr"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Select desired columns\n",
|
||||
"pumpkins <- pumpkins %>% \n",
|
||||
" select(Package, `Low Price`, `High Price`, Date)\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"# Print data set\n",
|
||||
"pumpkins %>% \n",
|
||||
" slice_head(n = 5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "F_FgxQnVZnM0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"#### dplyr::mutate()\n",
|
||||
"\n",
|
||||
"`mutate()` is a function in the package `dplyr` which helps you create or modify columns, while keeping the existing columns.\n",
|
||||
"\n",
|
||||
"The general structure of mutate is:\n",
|
||||
"\n",
|
||||
"`data %>% mutate(new_column_name = what_it_contains)`\n",
|
||||
"\n",
|
||||
"Let's take `mutate` out for a spin using the `Date` column by doing the following operations:\n",
|
||||
"\n",
|
||||
"1. Convert the dates (currently of type character) to a month format (these are US dates, so the format is `MM/DD/YYYY`).\n",
|
||||
"\n",
|
||||
"2. Extract the month from the dates to a new column.\n",
|
||||
"\n",
|
||||
"In R, the package [lubridate](https://lubridate.tidyverse.org/) makes it easier to work with Date-time data. So, let's use `dplyr::mutate()`, `lubridate::mdy()`, `lubridate::month()` and see how to achieve the above objectives. We can drop the Date column since we won't be needing it again in subsequent operations."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "2KKo0Ed9Z1VB"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Load lubridate\n",
|
||||
"library(lubridate)\n",
|
||||
"\n",
|
||||
"pumpkins <- pumpkins %>% \n",
|
||||
" # Convert the Date column to a date object\n",
|
||||
" mutate(Date = mdy(Date)) %>% \n",
|
||||
" # Extract month from Date\n",
|
||||
" mutate(Month = month(Date)) %>% \n",
|
||||
" # Drop Date column\n",
|
||||
" select(-Date)\n",
|
||||
"\n",
|
||||
"# View the first few rows\n",
|
||||
"pumpkins %>% \n",
|
||||
" slice_head(n = 7)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "5joszIVSZ6xe"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Woohoo! 🤩\n",
|
||||
"\n",
|
||||
"Next, let's create a new column `Price`, which represents the average price of a pumpkin. Now, let's take the average of the `Low Price` and `High Price` columns to populate the new Price column.\n",
|
||||
"<br>"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "nIgLjNMCZ-6Y"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Create a new column Price\n",
|
||||
"pumpkins <- pumpkins %>% \n",
|
||||
" mutate(Price = (`Low Price` + `High Price`)/2)\n",
|
||||
"\n",
|
||||
"# View the first few rows of the data\n",
|
||||
"pumpkins %>% \n",
|
||||
" slice_head(n = 5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "Zo0BsqqtaJw2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Yeees!💪\n",
|
||||
"\n",
|
||||
"\"But wait!\", you'll say after skimming through the whole data set with `View(pumpkins)`, \"There's something odd here!\"🤔\n",
|
||||
"\n",
|
||||
"If you look at the `Package` column, pumpkins are sold in many different configurations. Some are sold in `1 1/9 bushel` measures, and some in `1/2 bushel` measures, some per pumpkin, some per pound, and some in big boxes with varying widths.\n",
|
||||
"\n",
|
||||
"Let's verify this:"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "p77WZr-9aQAR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Verify the distinct observations in Package column\n",
|
||||
"pumpkins %>% \n",
|
||||
" distinct(Package)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "XISGfh0IaUy6"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Amazing!👏\n",
|
||||
"\n",
|
||||
"Pumpkins seem to be very hard to weigh consistently, so let's filter them by selecting only pumpkins with the string *bushel* in the `Package` column and put this in a new data frame `new_pumpkins`.\n",
|
||||
"<br>"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "7sMjiVujaZxY"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"#### dplyr::filter() and stringr::str_detect()\n",
|
||||
"\n",
|
||||
"[`dplyr::filter()`](https://dplyr.tidyverse.org/reference/filter.html): creates a subset of the data only containing **rows** that satisfy your conditions, in this case, pumpkins with the string *bushel* in the `Package` column.\n",
|
||||
"\n",
|
||||
"[stringr::str_detect()](https://stringr.tidyverse.org/reference/str_detect.html): detects the presence or absence of a pattern in a string.\n",
|
||||
"\n",
|
||||
"The [`stringr`](https://github.com/tidyverse/stringr) package provides simple functions for common string operations."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "L8Qfcs92ageF"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Retain only pumpkins with \"bushel\"\n",
|
||||
"new_pumpkins <- pumpkins %>% \n",
|
||||
" filter(str_detect(Package, \"bushel\"))\n",
|
||||
"\n",
|
||||
"# Get the dimensions of the new data\n",
|
||||
"dim(new_pumpkins)\n",
|
||||
"\n",
|
||||
"# View a few rows of the new data\n",
|
||||
"new_pumpkins %>% \n",
|
||||
" slice_head(n = 5)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "hy_SGYREampd"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"You can see that we have narrowed down to 415 or so rows of data containing pumpkins by the bushel.🤩\n",
|
||||
"<br>"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "VrDwF031avlR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"#### dplyr::case_when()\n",
|
||||
"\n",
|
||||
"**But wait! There's one more thing to do**\n",
|
||||
"\n",
|
||||
"Did you notice that the bushel amount varies per row? You need to normalize the pricing so that you show the pricing per bushel, not per 1 1/9 or 1/2 bushel. Time to do some math to standardize it.\n",
|
||||
"\n",
|
||||
"We'll use the function [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) to *mutate* the Price column depending on some conditions. `case_when` allows you to vectorise multiple `if_else()`statements.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mLpw2jH4a0tx"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Convert the price if the Package contains fractional bushel values\n",
|
||||
"new_pumpkins <- new_pumpkins %>% \n",
|
||||
" mutate(Price = case_when(\n",
|
||||
" str_detect(Package, \"1 1/9\") ~ Price/(1 + 1/9),\n",
|
||||
" str_detect(Package, \"1/2\") ~ Price/(1/2),\n",
|
||||
" TRUE ~ Price))\n",
|
||||
"\n",
|
||||
"# View the first few rows of the data\n",
|
||||
"new_pumpkins %>% \n",
|
||||
" slice_head(n = 30)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "P68kLVQmbM6I"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now, we can analyze the pricing per unit based on their bushel measurement. All this study of bushels of pumpkins, however, goes to show how very `important` it is to `understand the nature of your data`!\n",
|
||||
"\n",
|
||||
"> ✅ According to [The Spruce Eats](https://www.thespruceeats.com/how-much-is-a-bushel-1389308), a bushel's weight depends on the type of produce, as it's a volume measurement. \"A bushel of tomatoes, for example, is supposed to weigh 56 pounds... Leaves and greens take up more space with less weight, so a bushel of spinach is only 20 pounds.\" It's all pretty complicated! Let's not bother with making a bushel-to-pound conversion, and instead price by the bushel. All this study of bushels of pumpkins, however, goes to show how very important it is to understand the nature of your data!\n",
|
||||
">\n",
|
||||
"> ✅ Did you notice that pumpkins sold by the half-bushel are very expensive? Can you figure out why? Hint: little pumpkins are way pricier than big ones, probably because there are so many more of them per bushel, given the unused space taken by one big hollow pie pumpkin.\n",
|
||||
"<br>\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "pS2GNPagbSdb"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Now lastly, for the sheer sake of adventure 💁♀️, let's also move the Month column to the first position i.e `before` column `Package`.\n",
|
||||
"\n",
|
||||
"`dplyr::relocate()` is used to change column positions."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "qql1SowfbdnP"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Create a new data frame new_pumpkins\n",
|
||||
"new_pumpkins <- new_pumpkins %>% \n",
|
||||
" relocate(Month, .before = Package)\n",
|
||||
"\n",
|
||||
"new_pumpkins %>% \n",
|
||||
" slice_head(n = 7)"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "JJ1x6kw8bixF"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Good job!👌 You now have a clean, tidy dataset on which you can build your new regression model!\n",
|
||||
"<br>"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "y8TJ0Za_bn5Y"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## 4. Data visualization with ggplot2\n",
|
||||
"\n",
|
||||
"<p >\n",
|
||||
" <img src=\"../../images/data-visualization.png\"\n",
|
||||
" width=\"600\"/>\n",
|
||||
" <figcaption>Infographic by Dasani Madipalli</figcaption>\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"<!--{width=\"600\"}-->\n",
|
||||
"\n",
|
||||
"There is a *wise* saying that goes like this:\n",
|
||||
"\n",
|
||||
"> \"The simple graph has brought more information to the data analyst's mind than any other device.\" --- John Tukey\n",
|
||||
"\n",
|
||||
"Part of the data scientist's role is to demonstrate the quality and nature of the data they are working with. To do this, they often create interesting visualizations, or plots, graphs, and charts, showing different aspects of data. In this way, they are able to visually show relationships and gaps that are otherwise hard to uncover.\n",
|
||||
"\n",
|
||||
"Visualizations can also help determine the machine learning technique most appropriate for the data. A scatterplot that seems to follow a line, for example, indicates that the data is a good candidate for a linear regression exercise.\n",
|
||||
"\n",
|
||||
"R offers a number of several systems for making graphs, but [`ggplot2`](https://ggplot2.tidyverse.org/index.html) is one of the most elegant and most versatile. `ggplot2` allows you to compose graphs by **combining independent components**.\n",
|
||||
"\n",
|
||||
"Let's start with a simple scatter plot for the Price and Month columns.\n",
|
||||
"\n",
|
||||
"So in this case, we'll start with [`ggplot()`](https://ggplot2.tidyverse.org/reference/ggplot.html), supply a dataset and aesthetic mapping (with [`aes()`](https://ggplot2.tidyverse.org/reference/aes.html)) then add a layers (like [`geom_point()`](https://ggplot2.tidyverse.org/reference/geom_point.html)) for scatter plots.\n"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "mYSH6-EtbvNa"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Set a theme for the plots\n",
|
||||
"theme_set(theme_light())\n",
|
||||
"\n",
|
||||
"# Create a scatter plot\n",
|
||||
"p <- ggplot(data = new_pumpkins, aes(x = Price, y = Month))\n",
|
||||
"p + geom_point()"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "g2YjnGeOcLo4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Is this a useful plot 🤷? Does anything about it surprise you?\n",
|
||||
"\n",
|
||||
"It's not particularly useful as all it does is display in your data as a spread of points in a given month.\n",
|
||||
"<br>"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Ml7SDCLQcPvE"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"### **How do we make it useful?**\n",
|
||||
"\n",
|
||||
"To get charts to display useful data, you usually need to group the data somehow. For instance in our case, finding the average price of pumpkins for each month would provide more insights to the underlying patterns in our data. This leads us to one more **dplyr** flyby:\n",
|
||||
"\n",
|
||||
"#### `dplyr::group_by() %>% summarize()`\n",
|
||||
"\n",
|
||||
"Grouped aggregation in R can be easily computed using\n",
|
||||
"\n",
|
||||
"`dplyr::group_by() %>% summarize()`\n",
|
||||
"\n",
|
||||
"- `dplyr::group_by()` changes the unit of analysis from the complete dataset to individual groups such as per month.\n",
|
||||
"\n",
|
||||
"- `dplyr::summarize()` creates a new data frame with one column for each grouping variable and one column for each of the summary statistics that you have specified.\n",
|
||||
"\n",
|
||||
"For example, we can use the `dplyr::group_by() %>% summarize()` to group the pumpkins into groups based on the **Month** columns and then find the **mean price** for each month."
|
||||
],
|
||||
"metadata": {
|
||||
"id": "jMakvJZIcVkh"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Find the average price of pumpkins per month\r\n",
|
||||
"new_pumpkins %>%\r\n",
|
||||
" group_by(Month) %>% \r\n",
|
||||
" summarise(mean_price = mean(Price))"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "6kVSUa2Bcilf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"Succinct!✨\n",
|
||||
"\n",
|
||||
"Categorical features such as months are better represented using a bar plot 📊. The layers responsible for bar charts are `geom_bar()` and `geom_col()`. Consult `?geom_bar` to find out more.\n",
|
||||
"\n",
|
||||
"Let's whip up one!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "Kds48GUBcj3W"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"source": [
|
||||
"# Find the average price of pumpkins per month then plot a bar chart\r\n",
|
||||
"new_pumpkins %>%\r\n",
|
||||
" group_by(Month) %>% \r\n",
|
||||
" summarise(mean_price = mean(Price)) %>% \r\n",
|
||||
" ggplot(aes(x = Month, y = mean_price)) +\r\n",
|
||||
" geom_col(fill = \"midnightblue\", alpha = 0.7) +\r\n",
|
||||
" ylab(\"Pumpkin Price\")"
|
||||
],
|
||||
"outputs": [],
|
||||
"metadata": {
|
||||
"id": "VNbU1S3BcrxO"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"🤩🤩This is a more useful data visualization! It seems to indicate that the highest price for pumpkins occurs in September and October. Does that meet your expectation? Why or why not?\n",
|
||||
"\n",
|
||||
"Congratulations on finishing the second lesson 👏! You prepared your data for model building, then uncovered more insights using visualizations!"
|
||||
],
|
||||
"metadata": {
|
||||
"id": "zDm0VOzzcuzR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,345 @@
|
||||
---
|
||||
title: 'Build a regression model: prepare and visualize data'
|
||||
output:
|
||||
html_document:
|
||||
df_print: paged
|
||||
theme: flatly
|
||||
highlight: breezedark
|
||||
toc: yes
|
||||
toc_float: yes
|
||||
code_download: yes
|
||||
---
|
||||
|
||||
## **Linear Regression for Pumpkins - Lesson 2**
|
||||
|
||||
#### Introduction
|
||||
|
||||
Now that you are set up with the tools you need to start tackling machine learning model building with Tidymodels and the Tidyverse, you are ready to start asking questions of your data. As you work with data and apply ML solutions, it's very important to understand how to ask the right question to properly unlock the potentials of your dataset.
|
||||
|
||||
In this lesson, you will learn:
|
||||
|
||||
- How to prepare your data for model-building.
|
||||
|
||||
- How to use `ggplot2` for data visualization.
|
||||
|
||||
The question you need answered will determine what type of ML algorithms you will leverage. And the quality of the answer you get back will be heavily dependent on the nature of your data.
|
||||
|
||||
Let's see this by working through a practical exercise.
|
||||
|
||||
{width="700"}
|
||||
|
||||
## 1. Importing pumpkins data and summoning the Tidyverse
|
||||
|
||||
We'll require the following packages to slice and dice this lesson:
|
||||
|
||||
- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!
|
||||
|
||||
You can have them installed as:
|
||||
|
||||
`install.packages(c("tidyverse"))`
|
||||
|
||||
The script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.
|
||||
|
||||
```{r, message=F, warning=F}
|
||||
if (!require("pacman")) install.packages("pacman")
|
||||
pacman::p_load(tidyverse)
|
||||
```
|
||||
|
||||
Now, let's fire up some packages and load the [data](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/data/US-pumpkins.csv) provided for this lesson!
|
||||
|
||||
```{r load_tidy_verse_models, message=F, warning=F}
|
||||
# Load the core Tidyverse packages
|
||||
library(tidyverse)
|
||||
|
||||
# Import the pumpkins data
|
||||
pumpkins <- read_csv(file = "https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/2-Regression/data/US-pumpkins.csv")
|
||||
|
||||
|
||||
# Get a glimpse and dimensions of the data
|
||||
glimpse(pumpkins)
|
||||
|
||||
|
||||
# Print the first 50 rows of the data set
|
||||
pumpkins %>%
|
||||
slice_head(n =50)
|
||||
|
||||
```
|
||||
|
||||
A quick `glimpse()` immediately shows that there are blanks and a mix of strings (`chr`) and numeric data (`dbl`). The `Date` is of type character and there's also a strange column called `Package` where the data is a mix between `sacks`, `bins` and other values. The data, in fact, is a bit of a mess 😤.
|
||||
|
||||
In fact, it is not very common to be gifted a dataset that is completely ready to use to create a ML model out of the box. But worry not, in this lesson, you will learn how to prepare a raw dataset using standard R libraries 🧑🔧. You will also learn various techniques to visualize the data.📈📊
|
||||
|
||||
|
||||
|
||||
> A refresher: The pipe operator (`%>%`) performs operations in logical sequence by passing an object forward into a function or call expression. You can think of the pipe operator as saying "and then" in your code.
|
||||
|
||||
|
||||
## 2. Check for missing data
|
||||
|
||||
One of the most common issues data scientists need to deal with is incomplete or missing data. R represents missing, or unknown values, with special sentinel value: `NA` (Not Available).
|
||||
|
||||
So how would we know that the data frame contains missing values?
|
||||
|
||||
- One straight forward way would be to use the base R function `anyNA` which returns the logical objects `TRUE` or `FALSE`
|
||||
|
||||
```{r anyNA, message=F, warning=F}
|
||||
pumpkins %>%
|
||||
anyNA()
|
||||
```
|
||||
|
||||
Great, there seems to be some missing data! That's a good place to start.
|
||||
|
||||
- Another way would be to use the function `is.na()` that indicates which individual column elements are missing with a logical `TRUE`.
|
||||
|
||||
```{r is_na, message=F, warning=F}
|
||||
pumpkins %>%
|
||||
is.na() %>%
|
||||
head(n = 7)
|
||||
```
|
||||
|
||||
Okay, got the job done but with a large data frame such as this, it would be inefficient and practically impossible to review all of the rows and columns individually😴.
|
||||
|
||||
- A more intuitive way would be to calculate the sum of the missing values for each column:
|
||||
|
||||
```{r colSum_NA, message=F, warning=F}
|
||||
pumpkins %>%
|
||||
is.na() %>%
|
||||
colSums()
|
||||
```
|
||||
|
||||
Much better! There is missing data, but maybe it won't matter for the task at hand. Let's see what further analysis brings forth.
|
||||
|
||||
> Along with the awesome sets of packages and functions, R has a very good documentation. For instance, use `help(colSums)` or `?colSums` to find out more about the function.
|
||||
|
||||
## 3. Dplyr: A Grammar of Data Manipulation
|
||||
|
||||
{width="569"}
|
||||
|
||||
[`dplyr`](https://dplyr.tidyverse.org/), a package in the Tidyverse, is a grammar of data manipulation that provides a consistent set of verbs that help you solve the most common data manipulation challenges. In this section, we'll explore some of dplyr's verbs!
|
||||
|
||||
#### dplyr::select()
|
||||
|
||||
`select()` is a function in the package `dplyr` which helps you pick columns to keep or exclude.
|
||||
|
||||
To make your data frame easier to work with, drop several of its columns, using `select()`, keeping only the columns you need.
|
||||
|
||||
For instance, in this exercise, our analysis will involve the columns `Package`, `Low Price`, `High Price` and `Date`. Let's select these columns.
|
||||
|
||||
```{r select, message=F, warning=F}
|
||||
# Select desired columns
|
||||
pumpkins <- pumpkins %>%
|
||||
select(Package, `Low Price`, `High Price`, Date)
|
||||
|
||||
|
||||
# Print data set
|
||||
pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
```
|
||||
|
||||
#### dplyr::mutate()
|
||||
|
||||
`mutate()` is a function in the package `dplyr` which helps you create or modify columns, while keeping the existing columns.
|
||||
|
||||
The general structure of mutate is:
|
||||
|
||||
`data %>% mutate(new_column_name = what_it_contains)`
|
||||
|
||||
Let's take `mutate` out for a spin using the `Date` column by doing the following operations:
|
||||
|
||||
1. Convert the dates (currently of type character) to a month format (these are US dates, so the format is `MM/DD/YYYY`).
|
||||
|
||||
2. Extract the month from the dates to a new column.
|
||||
|
||||
In R, the package [lubridate](https://lubridate.tidyverse.org/) makes it easier to work with Date-time data. So, let's use `dplyr::mutate()`, `lubridate::mdy()`, `lubridate::month()` and see how to achieve the above objectives. We can drop the Date column since we won't be needing it again in subsequent operations.
|
||||
|
||||
```{r mut_date, message=F, warning=F}
|
||||
# Load lubridate
|
||||
library(lubridate)
|
||||
|
||||
pumpkins <- pumpkins %>%
|
||||
# Convert the Date column to a date object
|
||||
mutate(Date = mdy(Date)) %>%
|
||||
# Extract month from Date
|
||||
mutate(Month = month(Date)) %>%
|
||||
# Drop Date column
|
||||
select(-Date)
|
||||
|
||||
# View the first few rows
|
||||
pumpkins %>%
|
||||
slice_head(n = 7)
|
||||
```
|
||||
|
||||
Woohoo! 🤩
|
||||
|
||||
Next, let's create a new column `Price`, which represents the average price of a pumpkin. Now, let's take the average of the `Low Price` and `High Price` columns to populate the new Price column.
|
||||
|
||||
```{r price, message=F, warning=F}
|
||||
# Create a new column Price
|
||||
pumpkins <- pumpkins %>%
|
||||
mutate(Price = (`Low Price` + `High Price`)/2)
|
||||
|
||||
# View the first few rows of the data
|
||||
pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
```
|
||||
|
||||
Yeees!💪
|
||||
|
||||
"But wait!", you'll say after skimming through the whole data set with `View(pumpkins)`, "There's something odd here!"🤔
|
||||
|
||||
If you look at the `Package` column, pumpkins are sold in many different configurations. Some are sold in `1 1/9 bushel` measures, and some in `1/2 bushel` measures, some per pumpkin, some per pound, and some in big boxes with varying widths.
|
||||
|
||||
Let's verify this:
|
||||
|
||||
```{r Package, message=F, warning=F}
|
||||
# Verify the distinct observations in Package column
|
||||
pumpkins %>%
|
||||
distinct(Package)
|
||||
|
||||
```
|
||||
|
||||
Amazing!👏
|
||||
|
||||
Pumpkins seem to be very hard to weigh consistently, so let's filter them by selecting only pumpkins with the string *bushel* in the `Package` column and put this in a new data frame `new_pumpkins`.
|
||||
|
||||
#### dplyr::filter() and stringr::str_detect()
|
||||
|
||||
[`dplyr::filter()`](https://dplyr.tidyverse.org/reference/filter.html): creates a subset of the data only containing **rows** that satisfy your conditions, in this case, pumpkins with the string *bushel* in the `Package` column.
|
||||
|
||||
[stringr::str_detect()](https://stringr.tidyverse.org/reference/str_detect.html): detects the presence or absence of a pattern in a string.
|
||||
|
||||
The [`stringr`](https://github.com/tidyverse/stringr) package provides simple functions for common string operations.
|
||||
|
||||
```{r filter, message=F, warning=F}
|
||||
# Retain only pumpkins with "bushel"
|
||||
new_pumpkins <- pumpkins %>%
|
||||
filter(str_detect(Package, "bushel"))
|
||||
|
||||
# Get the dimensions of the new data
|
||||
dim(new_pumpkins)
|
||||
|
||||
# View a few rows of the new data
|
||||
new_pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
```
|
||||
|
||||
You can see that we have narrowed down to 415 or so rows of data containing pumpkins by the bushel.🤩
|
||||
|
||||
#### dplyr::case_when()
|
||||
|
||||
**But wait! There's one more thing to do**
|
||||
|
||||
Did you notice that the bushel amount varies per row? You need to normalize the pricing so that you show the pricing per bushel, not per 1 1/9 or 1/2 bushel. Time to do some math to standardize it.
|
||||
|
||||
We'll use the function [`case_when()`](https://dplyr.tidyverse.org/reference/case_when.html) to *mutate* the Price column depending on some conditions. `case_when` allows you to vectorise multiple `if_else()`statements.
|
||||
|
||||
```{r normalize_price, message=F, warning=F}
|
||||
# Convert the price if the Package contains fractional bushel values
|
||||
new_pumpkins <- new_pumpkins %>%
|
||||
mutate(Price = case_when(
|
||||
str_detect(Package, "1 1/9") ~ Price/(1 + 1/9),
|
||||
str_detect(Package, "1/2") ~ Price/(1/2),
|
||||
TRUE ~ Price))
|
||||
|
||||
# View the first few rows of the data
|
||||
new_pumpkins %>%
|
||||
slice_head(n = 30)
|
||||
```
|
||||
|
||||
Now, we can analyze the pricing per unit based on their bushel measurement. All this study of bushels of pumpkins, however, goes to show how very `important` it is to `understand the nature of your data`!
|
||||
|
||||
> ✅ According to [The Spruce Eats](https://www.thespruceeats.com/how-much-is-a-bushel-1389308), a bushel's weight depends on the type of produce, as it's a volume measurement. "A bushel of tomatoes, for example, is supposed to weigh 56 pounds... Leaves and greens take up more space with less weight, so a bushel of spinach is only 20 pounds." It's all pretty complicated! Let's not bother with making a bushel-to-pound conversion, and instead price by the bushel. All this study of bushels of pumpkins, however, goes to show how very important it is to understand the nature of your data!
|
||||
>
|
||||
> ✅ Did you notice that pumpkins sold by the half-bushel are very expensive? Can you figure out why? Hint: little pumpkins are way pricier than big ones, probably because there are so many more of them per bushel, given the unused space taken by one big hollow pie pumpkin.
|
||||
|
||||
Now lastly, for the sheer sake of adventure 💁♀️, let's also move the Month column to the first position i.e `before` column `Package`.
|
||||
|
||||
`dplyr::relocate()` is used to change column positions.
|
||||
|
||||
```{r new_pumpkins, message=F, warning=F}
|
||||
# Create a new data frame new_pumpkins
|
||||
new_pumpkins <- new_pumpkins %>%
|
||||
relocate(Month, .before = Package)
|
||||
|
||||
new_pumpkins %>%
|
||||
slice_head(n = 7)
|
||||
|
||||
```
|
||||
|
||||
Good job!👌 You now have a clean, tidy dataset on which you can build your new regression model!
|
||||
|
||||
## 4. Data visualization with ggplot2
|
||||
|
||||
{width="600"}
|
||||
|
||||
There is a *wise* saying that goes like this:
|
||||
|
||||
> "The simple graph has brought more information to the data analyst's mind than any other device." --- John Tukey
|
||||
|
||||
Part of the data scientist's role is to demonstrate the quality and nature of the data they are working with. To do this, they often create interesting visualizations, or plots, graphs, and charts, showing different aspects of data. In this way, they are able to visually show relationships and gaps that are otherwise hard to uncover.
|
||||
|
||||
Visualizations can also help determine the machine learning technique most appropriate for the data. A scatterplot that seems to follow a line, for example, indicates that the data is a good candidate for a linear regression exercise.
|
||||
|
||||
R offers a number of several systems for making graphs, but [`ggplot2`](https://ggplot2.tidyverse.org/index.html) is one of the most elegant and most versatile. `ggplot2` allows you to compose graphs by **combining independent components**.
|
||||
|
||||
Let's start with a simple scatter plot for the Price and Month columns.
|
||||
|
||||
So in this case, we'll start with [`ggplot()`](https://ggplot2.tidyverse.org/reference/ggplot.html), supply a dataset and aesthetic mapping (with [`aes()`](https://ggplot2.tidyverse.org/reference/aes.html)) then add a layers (like [`geom_point()`](https://ggplot2.tidyverse.org/reference/geom_point.html)) for scatter plots.
|
||||
|
||||
```{r scatter_plt, message=F, warning=F}
|
||||
# Set a theme for the plots
|
||||
theme_set(theme_light())
|
||||
|
||||
# Create a scatter plot
|
||||
p <- ggplot(data = new_pumpkins, aes(x = Price, y = Month))
|
||||
p + geom_point()
|
||||
```
|
||||
|
||||
Is this a useful plot 🤷? Does anything about it surprise you?
|
||||
|
||||
It's not particularly useful as all it does is display in your data as a spread of points in a given month.
|
||||
|
||||
### **How do we make it useful?**
|
||||
|
||||
To get charts to display useful data, you usually need to group the data somehow. For instance in our case, finding the average price of pumpkins for each month would provide more insights to the underlying patterns in our data. This leads us to one more **dplyr** flyby:
|
||||
|
||||
#### `dplyr::group_by() %>% summarize()`
|
||||
|
||||
Grouped aggregation in R can be easily computed using
|
||||
|
||||
`dplyr::group_by() %>% summarize()`
|
||||
|
||||
- `dplyr::group_by()` changes the unit of analysis from the complete dataset to individual groups such as per month.
|
||||
|
||||
- `dplyr::summarize()` creates a new data frame with one column for each grouping variable and one column for each of the summary statistics that you have specified.
|
||||
|
||||
For example, we can use the `dplyr::group_by() %>% summarize()` to group the pumpkins into groups based on the **Month** columns and then find the **mean price** for each month.
|
||||
|
||||
```{r grp_sumry, message=F, warning=F}
|
||||
# Find the average price of pumpkins per month
|
||||
new_pumpkins %>%
|
||||
group_by(Month) %>%
|
||||
summarise(mean_price = mean(Price))
|
||||
```
|
||||
|
||||
Succinct!✨
|
||||
|
||||
Categorical features such as months are better represented using a bar plot 📊. The layers responsible for bar charts are `geom_bar()` and `geom_col()`. Consult
|
||||
|
||||
`?geom_bar` to find out more.
|
||||
|
||||
Let's whip up one!
|
||||
|
||||
```{r bar_plt, message=F, warning=F}
|
||||
# Find the average price of pumpkins per month then plot a bar chart
|
||||
new_pumpkins %>%
|
||||
group_by(Month) %>%
|
||||
summarise(mean_price = mean(Price)) %>%
|
||||
ggplot(aes(x = Month, y = mean_price)) +
|
||||
geom_col(fill = "midnightblue", alpha = 0.7) +
|
||||
ylab("Pumpkin Price")
|
||||
```
|
||||
|
||||
🤩🤩This is a more useful data visualization! It seems to indicate that the highest price for pumpkins occurs in September and October. Does that meet your expectation? Why or why not?
|
||||
|
||||
Congratulations on finishing the second lesson 👏! You prepared your data for model building, then uncovered more insights using visualizations!
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,401 @@
|
||||
# Build a regression model using Scikit-learn: regression four ways
|
||||
|
||||
## Beginner Note
|
||||
|
||||
Linear regression is used when we want to predict a **numerical value** (for example, house price, temperature, or sales).
|
||||
It works by finding a straight line that best represents the relationship between input features and the output.
|
||||
|
||||
In this lesson, we focus on understanding the concept before exploring more advanced regression techniques.
|
||||
|
||||
> Infographic by [Dasani Madipalli](https://twitter.com/dasani_decoded)
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
> ### [This lesson is available in R!](./solution/R/lesson_3.html)
|
||||
### Introduction
|
||||
|
||||
So far you have explored what regression is with sample data gathered from the pumpkin pricing dataset that we will use throughout this lesson. You have also visualized it using Matplotlib.
|
||||
|
||||
Now you are ready to dive deeper into regression for ML. While visualization allows you to make sense of data, the real power of Machine Learning comes from _training models_. Models are trained on historic data to automatically capture data dependencies, and they allow you to predict outcomes for new data, which the model has not seem before.
|
||||
|
||||
In this lesson, you will learn more about two types of regression: _basic linear regression_ and _polynomial regression_, along with some of the math underlying these techniques. Those models will allow us to predict pumpkin prices depending on different input data.
|
||||
|
||||
[](https://youtu.be/CRxFT8oTDMg "ML for beginners - Understanding Linear Regression")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of linear regression.
|
||||
|
||||
> Throughout this curriculum, we assume minimal knowledge of math, and seek to make it accessible for students coming from other fields, so watch for notes, 🧮 callouts, diagrams, and other learning tools to aid in comprehension.
|
||||
|
||||
### Prerequisite
|
||||
|
||||
You should be familiar by now with the structure of the pumpkin data that we are examining. You can find it preloaded and pre-cleaned in this lesson's _notebook.ipynb_ file. In the file, the pumpkin price is displayed per bushel in a new data frame. Make sure you can run these notebooks in kernels in Visual Studio Code.
|
||||
|
||||
### Preparation
|
||||
|
||||
As a reminder, you are loading this data so as to ask questions of it.
|
||||
|
||||
- When is the best time to buy pumpkins?
|
||||
- What price can I expect of a case of miniature pumpkins?
|
||||
- Should I buy them in half-bushel baskets or by the 1 1/9 bushel box?
|
||||
Let's keep digging into this data.
|
||||
|
||||
In the previous lesson, you created a Pandas data frame and populated it with part of the original dataset, standardizing the pricing by the bushel. By doing that, however, you were only able to gather about 400 datapoints and only for the fall months.
|
||||
|
||||
Take a look at the data that we preloaded in this lesson's accompanying notebook. The data is preloaded and an initial scatterplot is charted to show month data. Maybe we can get a little more detail about the nature of the data by cleaning it more.
|
||||
|
||||
## A linear regression line
|
||||
|
||||
As you learned in Lesson 1, the goal of a linear regression exercise is to be able to plot a line to:
|
||||
|
||||
- **Show variable relationships**. Show the relationship between variables
|
||||
- **Make predictions**. Make accurate predictions on where a new datapoint would fall in relationship to that line.
|
||||
|
||||
It is typical of **Least-Squares Regression** to draw this type of line. The term "Least-Squares" refers to the process of minimizing the total error in our model. For every data point, we measure the vertical distance (called a residual) between the actual point and our regression line.
|
||||
|
||||
We square these distances for two main reasons:
|
||||
|
||||
1. **Magnitude over Direction:** We want to treat an error of -5 the same as an error of +5. Squaring turns all values positive.
|
||||
|
||||
2. **Penalizing Outliers:** Squaring gives more weight to larger errors, forcing the line to stay closer to points that are far away.
|
||||
|
||||
We then add all these squared values together. Our goal is to find the specific line where this final sum is at its least (the smallest possible value)—hence the name "Least-Squares".
|
||||
|
||||
> **🧮 Show me the math**
|
||||
>
|
||||
> This line, called the _line of best fit_ can be expressed by [an equation](https://en.wikipedia.org/wiki/Simple_linear_regression):
|
||||
>
|
||||
> ```
|
||||
> Y = a + bX
|
||||
> ```
|
||||
>
|
||||
> `X` is the 'explanatory variable'. `Y` is the 'dependent variable'. The slope of the line is `b` and `a` is the y-intercept, which refers to the value of `Y` when `X = 0`.
|
||||
>
|
||||
>
|
||||
>
|
||||
> First, calculate the slope `b`. Infographic by [Jen Looper](https://twitter.com/jenlooper)
|
||||
>
|
||||
> In other words, and referring to our pumpkin data's original question: "predict the price of a pumpkin per bushel by month", `X` would refer to the price and `Y` would refer to the month of sale.
|
||||
>
|
||||
>
|
||||
>
|
||||
> Calculate the value of Y. If you're paying around $4, it must be April! Infographic by [Jen Looper](https://twitter.com/jenlooper)
|
||||
>
|
||||
> The math that calculates the line must demonstrate the slope of the line, which is also dependent on the intercept, or where `Y` is situated when `X = 0`.
|
||||
>
|
||||
> You can observe the method of calculation for these values on the [Math is Fun](https://www.mathsisfun.com/data/least-squares-regression.html) web site. Also visit [this Least-squares calculator](https://www.mathsisfun.com/data/least-squares-calculator.html) to watch how the numbers' values impact the line.
|
||||
|
||||
## Correlation
|
||||
|
||||
One more term to understand is the **Correlation Coefficient** between given X and Y variables. Using a scatterplot, you can quickly visualize this coefficient. A plot with datapoints scattered in a neat line have high correlation, but a plot with datapoints scattered everywhere between X and Y have a low correlation.
|
||||
|
||||
A good linear regression model will be one that has a high (nearer to 1 than 0) Correlation Coefficient using the Least-Squares Regression method with a line of regression.
|
||||
|
||||
✅ Run the notebook accompanying this lesson and look at the Month to Price scatterplot. Does the data associating Month to Price for pumpkin sales seem to have high or low correlation, according to your visual interpretation of the scatterplot? Does that change if you use more fine-grained measure instead of `Month`, eg. *day of the year* (i.e. number of days since the beginning of the year)?
|
||||
|
||||
In the code below, we will assume that we have cleaned up the data, and obtained a data frame called `new_pumpkins`, similar to the following:
|
||||
|
||||
ID | Month | DayOfYear | Variety | City | Package | Low Price | High Price | Price
|
||||
---|-------|-----------|---------|------|---------|-----------|------------|-------
|
||||
70 | 9 | 267 | PIE TYPE | BALTIMORE | 1 1/9 bushel cartons | 15.0 | 15.0 | 13.636364
|
||||
71 | 9 | 267 | PIE TYPE | BALTIMORE | 1 1/9 bushel cartons | 18.0 | 18.0 | 16.363636
|
||||
72 | 10 | 274 | PIE TYPE | BALTIMORE | 1 1/9 bushel cartons | 18.0 | 18.0 | 16.363636
|
||||
73 | 10 | 274 | PIE TYPE | BALTIMORE | 1 1/9 bushel cartons | 17.0 | 17.0 | 15.454545
|
||||
74 | 10 | 281 | PIE TYPE | BALTIMORE | 1 1/9 bushel cartons | 15.0 | 15.0 | 13.636364
|
||||
|
||||
> The code to clean the data is available in [`notebook.ipynb`](notebook.ipynb). We have performed the same cleaning steps as in the previous lesson, and have calculated `DayOfYear` column using the following expression:
|
||||
|
||||
```python
|
||||
day_of_year = pd.to_datetime(pumpkins['Date']).apply(lambda dt: (dt-datetime(dt.year,1,1)).days)
|
||||
```
|
||||
|
||||
Now that you have an understanding of the math behind linear regression, let's create a Regression model to see if we can predict which package of pumpkins will have the best pumpkin prices. Someone buying pumpkins for a holiday pumpkin patch might want this information to be able to optimize their purchases of pumpkin packages for the patch.
|
||||
|
||||
## Looking for Correlation
|
||||
|
||||
[](https://youtu.be/uoRq-lW2eQo "ML for beginners - Looking for Correlation: The Key to Linear Regression")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of correlation.
|
||||
|
||||
From the previous lesson you have probably seen that the average price for different months looks like this:
|
||||
|
||||
<img alt="Average price by month" src="../2-Data/images/barchart.png" width="50%"/>
|
||||
|
||||
This suggests that there should be some correlation, and we can try training linear regression model to predict the relationship between `Month` and `Price`, or between `DayOfYear` and `Price`. Here is the scatter plot that shows the latter relationship:
|
||||
|
||||
<img alt="Scatter plot of Price vs. Day of Year" src="images/scatter-dayofyear.png" width="50%" />
|
||||
|
||||
Let's see if there is a correlation using the `corr` function:
|
||||
|
||||
```python
|
||||
print(new_pumpkins['Month'].corr(new_pumpkins['Price']))
|
||||
print(new_pumpkins['DayOfYear'].corr(new_pumpkins['Price']))
|
||||
```
|
||||
|
||||
It looks like the correlation is pretty small, -0.15 by `Month` and -0.17 by the `DayOfYear`, but there could be another important relationship. It looks like there are different clusters of prices corresponding to different pumpkin varieties. To confirm this hypothesis, let's plot each pumpkin category using a different color. By passing an `ax` parameter to the `scatter` plotting function we can plot all points on the same graph:
|
||||
|
||||
```python
|
||||
ax=None
|
||||
colors = ['red','blue','green','yellow']
|
||||
for i,var in enumerate(new_pumpkins['Variety'].unique()):
|
||||
df = new_pumpkins[new_pumpkins['Variety']==var]
|
||||
ax = df.plot.scatter('DayOfYear','Price',ax=ax,c=colors[i],label=var)
|
||||
```
|
||||
|
||||
<img alt="Scatter plot of Price vs. Day of Year" src="images/scatter-dayofyear-color.png" width="50%" />
|
||||
|
||||
Our investigation suggests that variety has more effect on the overall price than the actual selling date. We can see this with a bar graph:
|
||||
|
||||
```python
|
||||
new_pumpkins.groupby('Variety')['Price'].mean().plot(kind='bar')
|
||||
```
|
||||
|
||||
<img alt="Bar graph of price vs variety" src="images/price-by-variety.png" width="50%" />
|
||||
|
||||
Let us focus for the moment only on one pumpkin variety, the 'pie type', and see what effect the date has on the price:
|
||||
|
||||
```python
|
||||
pie_pumpkins = new_pumpkins[new_pumpkins['Variety']=='PIE TYPE']
|
||||
pie_pumpkins.plot.scatter('DayOfYear','Price')
|
||||
```
|
||||
<img alt="Scatter plot of Price vs. Day of Year" src="images/pie-pumpkins-scatter.png" width="50%" />
|
||||
|
||||
If we now calculate the correlation between `Price` and `DayOfYear` using `corr` function, we will get something like `-0.27` - which means that training a predictive model makes sense.
|
||||
|
||||
> Before training a linear regression model, it is important to make sure that our data is clean. Linear regression does not work well with missing values, thus it makes sense to get rid of all empty cells:
|
||||
|
||||
```python
|
||||
pie_pumpkins.dropna(inplace=True)
|
||||
pie_pumpkins.info()
|
||||
```
|
||||
|
||||
Another approach would be to fill those empty values with mean values from the corresponding column.
|
||||
|
||||
## Simple Linear Regression
|
||||
|
||||
[](https://youtu.be/e4c_UP2fSjg "ML for beginners - Linear and Polynomial Regression using Scikit-learn")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of linear and polynomial regression.
|
||||
|
||||
To train our Linear Regression model, we will use the **Scikit-learn** library.
|
||||
|
||||
```python
|
||||
from sklearn.linear_model import LinearRegression
|
||||
from sklearn.metrics import mean_squared_error
|
||||
from sklearn.model_selection import train_test_split
|
||||
```
|
||||
|
||||
We start by separating input values (features) and the expected output (label) into separate numpy arrays:
|
||||
|
||||
```python
|
||||
X = pie_pumpkins['DayOfYear'].to_numpy().reshape(-1,1)
|
||||
y = pie_pumpkins['Price']
|
||||
```
|
||||
|
||||
> Note that we had to perform `reshape` on the input data in order for the Linear Regression package to understand it correctly. Linear Regression expects a 2D-array as an input, where each row of the array corresponds to a vector of input features. In our case, since we have only one input - we need an array with shape N×1, where N is the dataset size.
|
||||
|
||||
Then, we need to split the data into train and test datasets, so that we can validate our model after training:
|
||||
|
||||
```python
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
|
||||
```
|
||||
|
||||
Finally, training the actual Linear Regression model takes only two lines of code. We define the `LinearRegression` object, and fit it to our data using the `fit` method:
|
||||
|
||||
```python
|
||||
lin_reg = LinearRegression()
|
||||
lin_reg.fit(X_train,y_train)
|
||||
```
|
||||
|
||||
The `LinearRegression` object after `fit`-ting contains all the coefficients of the regression, which can be accessed using `.coef_` property. In our case, there is just one coefficient, which should be around `-0.017`. It means that prices seem to drop a bit with time, but not too much, around 2 cents per day. We can also access the intersection point of the regression with Y-axis using `lin_reg.intercept_` - it will be around `21` in our case, indicating the price at the beginning of the year.
|
||||
|
||||
To see how accurate our model is, we can predict prices on a test dataset, and then measure how close our predictions are to the expected values. This can be done using root mean square error (RMSE) metrics, which is the root of the mean of all squared differences between expected and predicted value.
|
||||
|
||||
```python
|
||||
pred = lin_reg.predict(X_test)
|
||||
|
||||
rmse = np.sqrt(mean_squared_error(y_test,pred))
|
||||
print(f'RMSE: {rmse:3.3} ({rmse/np.mean(pred)*100:3.3}%)')
|
||||
```
|
||||
|
||||
Our error seems to be around 2 points, which is ~17%. Not too good. Another indicator of model quality is the **coefficient of determination**, which can be obtained like this:
|
||||
|
||||
```python
|
||||
score = lin_reg.score(X_train,y_train)
|
||||
print('Model determination: ', score)
|
||||
```
|
||||
If the value is 0, it means that the model does not take input data into account, and acts as the *worst linear predictor*, which is simply a mean value of the result. The value of 1 means that we can perfectly predict all expected outputs. In our case, the coefficient is around 0.06, which is quite low.
|
||||
|
||||
We can also plot the test data together with the regression line to better see how regression works in our case:
|
||||
|
||||
```python
|
||||
plt.scatter(X_test,y_test)
|
||||
plt.plot(X_test,pred)
|
||||
```
|
||||
|
||||
<img alt="Linear regression" src="images/linear-results.png" width="50%" />
|
||||
|
||||
## Polynomial Regression
|
||||
|
||||
Another type of Linear Regression is Polynomial Regression. While sometimes there's a linear relationship between variables - the bigger the pumpkin in volume, the higher the price - sometimes these relationships can't be plotted as a plane or straight line.
|
||||
|
||||
✅ Here are [some more examples](https://online.stat.psu.edu/stat501/lesson/9/9.8) of data that could use Polynomial Regression
|
||||
|
||||
Take another look at the relationship between Date and Price. Does this scatterplot seem like it should necessarily be analyzed by a straight line? Can't prices fluctuate? In this case, you can try polynomial regression.
|
||||
|
||||
✅ Polynomials are mathematical expressions that might consist of one or more variables and coefficients
|
||||
|
||||
Polynomial regression creates a curved line to better fit nonlinear data. In our case, if we include a squared `DayOfYear` variable into input data, we should be able to fit our data with a parabolic curve, which will have a minimum at a certain point within the year.
|
||||
|
||||
Scikit-learn includes a helpful [pipeline API](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.make_pipeline.html?highlight=pipeline#sklearn.pipeline.make_pipeline) to combine different steps of data processing together. A **pipeline** is a chain of **estimators**. In our case, we will create a pipeline that first adds polynomial features to our model, and then trains the regression:
|
||||
|
||||
```python
|
||||
from sklearn.preprocessing import PolynomialFeatures
|
||||
from sklearn.pipeline import make_pipeline
|
||||
|
||||
pipeline = make_pipeline(PolynomialFeatures(2), LinearRegression())
|
||||
|
||||
pipeline.fit(X_train,y_train)
|
||||
```
|
||||
|
||||
Using `PolynomialFeatures(2)` means that we will include all second-degree polynomials from the input data. In our case it will just mean `DayOfYear`<sup>2</sup>, but given two input variables X and Y, this will add X<sup>2</sup>, XY and Y<sup>2</sup>. We may also use higher degree polynomials if we want.
|
||||
|
||||
Pipelines can be used in the same manner as the original `LinearRegression` object, i.e. we can `fit` the pipeline, and then use `predict` to get the prediction results:
|
||||
|
||||
```python
|
||||
pred = pipeline.predict(X_test)
|
||||
|
||||
rmse = np.sqrt(mean_squared_error(y_test,pred))
|
||||
print(f'RMSE: {rmse:3.3} ({rmse/np.mean(pred)*100:3.3}%)')
|
||||
|
||||
score = pipeline.score(X_train,y_train)
|
||||
print('Model determination: ', score)
|
||||
```
|
||||
|
||||
To plot the smooth approximation curve, we use `np.linspace` to create a uniform range of input values, rather than plotting directly on the unordered test data (which would produce a zigzag line):
|
||||
|
||||
```python
|
||||
X_range = np.linspace(X_test.min(), X_test.max(), 100).reshape(-1,1)
|
||||
y_range = pipeline.predict(X_range)
|
||||
|
||||
plt.scatter(X_test, y_test)
|
||||
plt.plot(X_range, y_range)
|
||||
```
|
||||
|
||||
Here is the graph showing test data, and the approximation curve:
|
||||
|
||||
<img alt="Polynomial regression" src="images/poly-results.png" width="50%" />
|
||||
|
||||
Using Polynomial Regression, we can get slightly lower RMSE and higher determination, but not significantly. We need to take into account other features!
|
||||
|
||||
> You can see that the minimal pumpkin prices are observed somewhere around Halloween. How can you explain this?
|
||||
|
||||
🎃 Congratulations, you just created a model that can help predict the price of pie pumpkins. You can probably repeat the same procedure for all pumpkin types, but that would be tedious. Let's learn now how to take pumpkin variety into account in our model!
|
||||
|
||||
## Categorical Features
|
||||
|
||||
In the ideal world, we want to be able to predict prices for different pumpkin varieties using the same model. However, the `Variety` column is somewhat different from columns like `Month`, because it contains non-numeric values. Such columns are called **categorical**.
|
||||
|
||||
[](https://youtu.be/DYGliioIAE0 "ML for beginners - Categorical Feature Predictions with Linear Regression")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of using categorical features.
|
||||
|
||||
Here you can see how average price depends on variety:
|
||||
|
||||
<img alt="Average price by variety" src="images/price-by-variety.png" width="50%" />
|
||||
|
||||
To take variety into account, we first need to convert it to numeric form, or **encode** it. There are several way we can do it:
|
||||
|
||||
* Simple **numeric encoding** will build a table of different varieties, and then replace the variety name by an index in that table. This is not the best idea for linear regression, because linear regression takes the actual numeric value of the index, and adds it to the result, multiplying by some coefficient. In our case, the relationship between the index number and the price is clearly non-linear, even if we make sure that indices are ordered in some specific way.
|
||||
* **One-hot encoding** will replace the `Variety` column by 4 different columns, one for each variety. Each column will contain `1` if the corresponding row is of a given variety, and `0` otherwise. This means that there will be four coefficients in linear regression, one for each pumpkin variety, responsible for "starting price" (or rather "additional price") for that particular variety.
|
||||
|
||||
The code below shows how we can one-hot encode a variety:
|
||||
|
||||
```python
|
||||
pd.get_dummies(new_pumpkins['Variety'])
|
||||
```
|
||||
|
||||
ID | FAIRYTALE | MINIATURE | MIXED HEIRLOOM VARIETIES | PIE TYPE
|
||||
----|-----------|-----------|--------------------------|----------
|
||||
70 | 0 | 0 | 0 | 1
|
||||
71 | 0 | 0 | 0 | 1
|
||||
... | ... | ... | ... | ...
|
||||
1738 | 0 | 1 | 0 | 0
|
||||
1739 | 0 | 1 | 0 | 0
|
||||
1740 | 0 | 1 | 0 | 0
|
||||
1741 | 0 | 1 | 0 | 0
|
||||
1742 | 0 | 1 | 0 | 0
|
||||
|
||||
To train linear regression using one-hot encoded variety as input, we just need to initialize `X` and `y` data correctly:
|
||||
|
||||
```python
|
||||
X = pd.get_dummies(new_pumpkins['Variety'])
|
||||
y = new_pumpkins['Price']
|
||||
```
|
||||
|
||||
The rest of the code is the same as what we used above to train Linear Regression. If you try it, you will see that the mean squared error is about the same, but we get much higher coefficient of determination (~77%). To get even more accurate predictions, we can take more categorical features into account, as well as numeric features, such as `Month` or `DayOfYear`. To get one large array of features, we can use `join`:
|
||||
|
||||
```python
|
||||
X = pd.get_dummies(new_pumpkins['Variety']) \
|
||||
.join(new_pumpkins['Month']) \
|
||||
.join(pd.get_dummies(new_pumpkins['City'])) \
|
||||
.join(pd.get_dummies(new_pumpkins['Package']))
|
||||
y = new_pumpkins['Price']
|
||||
```
|
||||
|
||||
Here we also take into account `City` and `Package` type, which gives us RMSE 2.84 (10.5%), and determination 0.94!
|
||||
|
||||
## Putting it all together
|
||||
|
||||
To make the best model, we can use combined (one-hot encoded categorical + numeric) data from the above example together with Polynomial Regression. Here is the complete code for your convenience:
|
||||
|
||||
```python
|
||||
# set up training data
|
||||
X = pd.get_dummies(new_pumpkins['Variety']) \
|
||||
.join(new_pumpkins['Month']) \
|
||||
.join(pd.get_dummies(new_pumpkins['City'])) \
|
||||
.join(pd.get_dummies(new_pumpkins['Package']))
|
||||
y = new_pumpkins['Price']
|
||||
|
||||
# make train-test split
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
|
||||
|
||||
# setup and train the pipeline
|
||||
pipeline = make_pipeline(PolynomialFeatures(2), LinearRegression())
|
||||
pipeline.fit(X_train,y_train)
|
||||
|
||||
# predict results for test data
|
||||
pred = pipeline.predict(X_test)
|
||||
|
||||
# calculate RMSE and determination
|
||||
rmse = mean_squared_error(y_test, pred, squared=False)
|
||||
print(f'RMSE: {rmse:3.3} ({rmse/pred.mean()*100:3.3}%)')
|
||||
|
||||
score = pipeline.score(X_train,y_train)
|
||||
print('Model determination: ', score)
|
||||
```
|
||||
|
||||
This should give us the best determination coefficient of almost 97%, and RMSE=2.23 (~8% prediction error).
|
||||
|
||||
| Model | RMSE | Determination |
|
||||
|-------|-----|---------------|
|
||||
| `DayOfYear` Linear | 2.77 (17.2%) | 0.07 |
|
||||
| `DayOfYear` Polynomial | 2.73 (17.0%) | 0.08 |
|
||||
| `Variety` Linear | 5.24 (19.7%) | 0.77 |
|
||||
| All features Linear | 2.84 (10.5%) | 0.94 |
|
||||
| All features Polynomial | 2.23 (8.25%) | 0.97 |
|
||||
|
||||
🏆 Well done! You created four Regression models in one lesson, and improved the model quality to 97%. In the final section on Regression, you will learn about Logistic Regression to determine categories.
|
||||
|
||||
---
|
||||
## 🚀Challenge
|
||||
|
||||
Test several different variables in this notebook to see how correlation corresponds to model accuracy.
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
In this lesson we learned about Linear Regression. There are other important types of Regression. Read about Stepwise, Ridge, Lasso and Elasticnet techniques. A good course to study to learn more is the [Stanford Statistical Learning course](https://online.stanford.edu/courses/sohs-ystatslearning-statistical-learning)
|
||||
|
||||
## Assignment
|
||||
|
||||
[Build a Model](assignment.md)
|
||||
@@ -0,0 +1,11 @@
|
||||
# Create a Regression Model
|
||||
|
||||
## Instructions
|
||||
|
||||
In this lesson you were shown how to build a model using both Linear and Polynomial Regression. Using this knowledge, find a dataset or use one of Scikit-learn's built-in sets to build a fresh model. Explain in your notebook why you chose the technique you did, and demonstrate your model's accuracy. If it is not accurate, explain why.
|
||||
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | ------------------------------------------------------------ | -------------------------- | ------------------------------- |
|
||||
| | presents a complete notebook with a well-documented solution | the solution is incomplete | the solution is flawed or buggy |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 204 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 873 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 287 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 167 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.1 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 209 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 5.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 50 KiB |
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Pumpkin Pricing\n",
|
||||
"\n",
|
||||
"Load up required libraries and dataset. Convert the data to a dataframe containing a subset of the data: \n",
|
||||
"\n",
|
||||
"- Only get pumpkins priced by the bushel\n",
|
||||
"- Convert the date to a month\n",
|
||||
"- Calculate the price to be an average of high and low prices\n",
|
||||
"- Convert the price to reflect the pricing by bushel quantity"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import pandas as pd\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"from datetime import datetime\n",
|
||||
"\n",
|
||||
"pumpkins = pd.read_csv('../data/US-pumpkins.csv')\n",
|
||||
"\n",
|
||||
"pumpkins.head()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"pumpkins = pumpkins[pumpkins['Package'].str.contains('bushel', case=True, regex=True)]\n",
|
||||
"\n",
|
||||
"columns_to_select = ['Package', 'Variety', 'City Name', 'Low Price', 'High Price', 'Date']\n",
|
||||
"pumpkins = pumpkins.loc[:, columns_to_select]\n",
|
||||
"\n",
|
||||
"price = (pumpkins['Low Price'] + pumpkins['High Price']) / 2\n",
|
||||
"\n",
|
||||
"month = pd.DatetimeIndex(pumpkins['Date']).month\n",
|
||||
"day_of_year = pd.to_datetime(pumpkins['Date']).apply(lambda dt: (dt-datetime(dt.year,1,1)).days)\n",
|
||||
"\n",
|
||||
"new_pumpkins = pd.DataFrame(\n",
|
||||
" {'Month': month, \n",
|
||||
" 'DayOfYear' : day_of_year, \n",
|
||||
" 'Variety': pumpkins['Variety'], \n",
|
||||
" 'City': pumpkins['City Name'], \n",
|
||||
" 'Package': pumpkins['Package'], \n",
|
||||
" 'Low Price': pumpkins['Low Price'],\n",
|
||||
" 'High Price': pumpkins['High Price'], \n",
|
||||
" 'Price': price})\n",
|
||||
"\n",
|
||||
"new_pumpkins.loc[new_pumpkins['Package'].str.contains('1 1/9'), 'Price'] = price/1.1\n",
|
||||
"new_pumpkins.loc[new_pumpkins['Package'].str.contains('1/2'), 'Price'] = price*2\n",
|
||||
"\n",
|
||||
"new_pumpkins.head()\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"A basic scatterplot reminds us that we only have month data from August through December. We probably need more data to be able to draw conclusions in a linear fashion."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"plt.scatter('Month','Price',data=new_pumpkins)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"\n",
|
||||
"plt.scatter('DayOfYear','Price',data=new_pumpkins)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.8.3-final"
|
||||
},
|
||||
"orig_nbformat": 2
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
This is a temporary placeholder
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,679 @@
|
||||
---
|
||||
title: 'Build a regression model: linear and polynomial regression models'
|
||||
output:
|
||||
html_document:
|
||||
df_print: paged
|
||||
theme: flatly
|
||||
highlight: breezedark
|
||||
toc: yes
|
||||
toc_float: yes
|
||||
code_download: yes
|
||||
---
|
||||
|
||||
## Linear and Polynomial Regression for Pumpkin Pricing - Lesson 3
|
||||
|
||||
{width="800"}
|
||||
|
||||
#### Introduction
|
||||
|
||||
So far you have explored what regression is with sample data gathered from the pumpkin pricing dataset that we will use throughout this lesson. You have also visualized it using `ggplot2`.💪
|
||||
|
||||
Now you are ready to dive deeper into regression for ML. In this lesson, you will learn more about two types of regression: *basic linear regression* and *polynomial regression*, along with some of the math underlying these techniques.
|
||||
|
||||
> Throughout this curriculum, we assume minimal knowledge of math, and seek to make it accessible for students coming from other fields, so watch for notes, 🧮 callouts, diagrams, and other learning tools to aid in comprehension.
|
||||
|
||||
#### Preparation
|
||||
|
||||
As a reminder, you are loading this data so as to ask questions of it.
|
||||
|
||||
- When is the best time to buy pumpkins?
|
||||
|
||||
- What price can I expect of a case of miniature pumpkins?
|
||||
|
||||
- Should I buy them in half-bushel baskets or by the 1 1/9 bushel box? Let's keep digging into this data.
|
||||
|
||||
In the previous lesson, you created a `tibble` (a modern reimagining of the data frame) and populated it with part of the original dataset, standardizing the pricing by the bushel. By doing that, however, you were only able to gather about 400 data points and only for the fall months. Maybe we can get a little more detail about the nature of the data by cleaning it more? We'll see... 🕵️♀️
|
||||
|
||||
For this task, we'll require the following packages:
|
||||
|
||||
- `tidyverse`: The [tidyverse](https://www.tidyverse.org/) is a [collection of R packages](https://www.tidyverse.org/packages) designed to makes data science faster, easier and more fun!
|
||||
|
||||
- `tidymodels`: The [tidymodels](https://www.tidymodels.org/) framework is a [collection of packages](https://www.tidymodels.org/packages/) for modeling and machine learning.
|
||||
|
||||
- `janitor`: The [janitor package](https://github.com/sfirke/janitor) provides simple little tools for examining and cleaning dirty data.
|
||||
|
||||
- `corrplot`: The [corrplot package](https://cran.r-project.org/web/packages/corrplot/vignettes/corrplot-intro.html) provides a visual exploratory tool on correlation matrix that supports automatic variable reordering to help detect hidden patterns among variables.
|
||||
|
||||
You can have them installed as:
|
||||
|
||||
`install.packages(c("tidyverse", "tidymodels", "janitor", "corrplot"))`
|
||||
|
||||
The script below checks whether you have the packages required to complete this module and installs them for you in case they are missing.
|
||||
|
||||
```{r, message=F, warning=F}
|
||||
suppressWarnings(if (!require("pacman")) install.packages("pacman"))
|
||||
|
||||
pacman::p_load(tidyverse, tidymodels, janitor, corrplot)
|
||||
```
|
||||
|
||||
We'll later load these awesome packages and make them available in our current R session. (This is for mere illustration, `pacman::p_load()` already did that for you)
|
||||
|
||||
## 1. A linear regression line
|
||||
|
||||
As you learned in Lesson 1, the goal of a linear regression exercise is to be able to plot a *line* *of* *best fit* to:
|
||||
|
||||
- **Show variable relationships**. Show the relationship between variables
|
||||
|
||||
- **Make predictions**. Make accurate predictions on where a new data point would fall in relationship to that line.
|
||||
|
||||
To draw this type of line, we use a statistical technique called **Least-Squares Regression**. The term `least-squares` means that all the data points surrounding the regression line are squared and then added up. Ideally, that final sum is as small as possible, because we want a low number of errors, or `least-squares`. As such, the line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.
|
||||
|
||||
We do so since we want to model a line that has the least cumulative distance from all of our data points. We also square the terms before adding them since we are concerned with its magnitude rather than its direction.
|
||||
|
||||
> **🧮 Show me the math**
|
||||
>
|
||||
> This line, called the *line of best fit* can be expressed by [an equation](https://en.wikipedia.org/wiki/Simple_linear_regression):
|
||||
>
|
||||
> Y = a + bX
|
||||
>
|
||||
> `X` is the '`explanatory variable` or `predictor`'. `Y` is the '`dependent variable` or `outcome`'. The slope of the line is `b` and `a` is the y-intercept, which refers to the value of `Y` when `X = 0`.
|
||||
>
|
||||
> {width="400"}
|
||||
>
|
||||
> First, calculate the slope `b`.
|
||||
>
|
||||
> In other words, and referring to our pumpkin data's original question: "predict the price of a pumpkin per bushel by month", `X` would refer to the price and `Y` would refer to the month of sale.
|
||||
>
|
||||
|
||||
|
||||
> Calculate the value of Y. If you're paying around \$4, it must be April!
|
||||
>
|
||||
> The math that calculates the line must demonstrate the slope of the line, which is also dependent on the intercept, or where `Y` is situated when `X = 0`.
|
||||
>
|
||||
> You can observe the method of calculation for these values on the [Math is Fun](https://www.mathsisfun.com/data/least-squares-regression.html) web site. Also visit [this Least-squares calculator](https://www.mathsisfun.com/data/least-squares-calculator.html) to watch how the numbers' values impact the line.
|
||||
|
||||
Not so scary, right? 🤓
|
||||
|
||||
#### Correlation
|
||||
|
||||
One more term to understand is the **Correlation Coefficient** between given X and Y variables. Using a scatterplot, you can quickly visualize this coefficient. A plot with datapoints scattered in a neat line have high correlation, but a plot with datapoints scattered everywhere between X and Y have a low correlation.
|
||||
|
||||
A good linear regression model will be one that has a high (nearer to 1 than 0) Correlation Coefficient using the Least-Squares Regression method with a line of regression.
|
||||
|
||||
## **2. A dance with data: creating a data frame that will be used for modelling**
|
||||
|
||||
{width="700"}
|
||||
|
||||
Load up required libraries and dataset. Convert the data to a data frame containing a subset of the data:
|
||||
|
||||
- Only get pumpkins priced by the bushel
|
||||
|
||||
- Convert the date to a month
|
||||
|
||||
- Calculate the price to be an average of high and low prices
|
||||
|
||||
- Convert the price to reflect the pricing by bushel quantity
|
||||
|
||||
> We covered these steps in the [previous lesson](https://github.com/microsoft/ML-For-Beginners/blob/main/2-Regression/2-Data/solution/lesson_2.html).
|
||||
|
||||
```{r load_tidy_verse_models, message=F, warning=F}
|
||||
# Load the core Tidyverse packages
|
||||
library(tidyverse)
|
||||
library(lubridate)
|
||||
|
||||
# Import the pumpkins data
|
||||
pumpkins <- read_csv(file = "https://raw.githubusercontent.com/microsoft/ML-For-Beginners/main/2-Regression/data/US-pumpkins.csv")
|
||||
|
||||
|
||||
# Get a glimpse and dimensions of the data
|
||||
glimpse(pumpkins)
|
||||
|
||||
|
||||
# Print the first 50 rows of the data set
|
||||
pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
|
||||
```
|
||||
|
||||
In the spirit of sheer adventure, let's explore the [`janitor package`](github.com/sfirke/janitor) that provides simple functions for examining and cleaning dirty data. For instance, let's take a look at the column names for our data:
|
||||
|
||||
```{r col_names}
|
||||
# Return column names
|
||||
pumpkins %>%
|
||||
names()
|
||||
|
||||
```
|
||||
|
||||
🤔 We can do better. Let's make these column names `friendR` by converting them to the [snake_case](https://en.wikipedia.org/wiki/Snake_case) convention using `janitor::clean_names`. To find out more about this function: `?clean_names`
|
||||
|
||||
```{r friendR}
|
||||
# Clean names to the snake_case convention
|
||||
pumpkins <- pumpkins %>%
|
||||
clean_names(case = "snake")
|
||||
|
||||
# Return column names
|
||||
pumpkins %>%
|
||||
names()
|
||||
|
||||
```
|
||||
|
||||
Much tidyR 🧹! Now, a dance with the data using `dplyr` as in the previous lesson! 💃
|
||||
|
||||
```{r prep_data, message=F, warning=F}
|
||||
# Select desired columns
|
||||
pumpkins <- pumpkins %>%
|
||||
select(variety, city_name, package, low_price, high_price, date)
|
||||
|
||||
|
||||
|
||||
# Extract the month from the dates to a new column
|
||||
pumpkins <- pumpkins %>%
|
||||
mutate(date = mdy(date),
|
||||
month = month(date)) %>%
|
||||
select(-date)
|
||||
|
||||
|
||||
|
||||
# Create a new column for average Price
|
||||
pumpkins <- pumpkins %>%
|
||||
mutate(price = (low_price + high_price)/2)
|
||||
|
||||
|
||||
# Retain only pumpkins with the string "bushel"
|
||||
new_pumpkins <- pumpkins %>%
|
||||
filter(str_detect(string = package, pattern = "bushel"))
|
||||
|
||||
|
||||
# Normalize the pricing so that you show the pricing per bushel, not per 1 1/9 or 1/2 bushel
|
||||
new_pumpkins <- new_pumpkins %>%
|
||||
mutate(price = case_when(
|
||||
str_detect(package, "1 1/9") ~ price/(1.1),
|
||||
str_detect(package, "1/2") ~ price*2,
|
||||
TRUE ~ price))
|
||||
|
||||
# Relocate column positions
|
||||
new_pumpkins <- new_pumpkins %>%
|
||||
relocate(month, .before = variety)
|
||||
|
||||
|
||||
# Display the first 5 rows
|
||||
new_pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
```
|
||||
|
||||
Good job!👌 You now have a clean, tidy data set on which you can build your new regression model!
|
||||
|
||||
Mind a scatter plot?
|
||||
|
||||
```{r scatter_price_month}
|
||||
# Set theme
|
||||
theme_set(theme_light())
|
||||
|
||||
# Make a scatter plot of month and price
|
||||
new_pumpkins %>%
|
||||
ggplot(mapping = aes(x = month, y = price)) +
|
||||
geom_point(size = 1.6)
|
||||
|
||||
```
|
||||
|
||||
A scatter plot reminds us that we only have month data from August through December. We probably need more data to be able to draw conclusions in a linear fashion.
|
||||
|
||||
Let's take a look at our modelling data again:
|
||||
|
||||
```{r modelling data}
|
||||
# Display first 5 rows
|
||||
new_pumpkins %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
```
|
||||
|
||||
What if we wanted to predict the `price` of a pumpkin based on the `city` or `package` columns which are of type character? Or even more simply, how could we find the correlation (which requires both of its inputs to be numeric) between, say, `package` and `price`? 🤷🤷
|
||||
|
||||
Machine learning models work best with numeric features rather than text values, so you generally need to convert categorical features into numeric representations.
|
||||
|
||||
This means that we have to find a way to reformat our predictors to make them easier for a model to use effectively, a process known as `feature engineering`.
|
||||
|
||||
## 3. Preprocessing data for modelling with recipes 👩🍳👨🍳
|
||||
|
||||
Activities that reformat predictor values to make them easier for a model to use effectively has been termed `feature engineering`.
|
||||
|
||||
Different models have different preprocessing requirements. For instance, least squares requires `encoding categorical variables` such as month, variety and city_name. This simply involves `translating` a column with `categorical values` into one or more `numeric columns` that take the place of the original.
|
||||
|
||||
For example, suppose your data includes the following categorical feature:
|
||||
|
||||
| city |
|
||||
|:-------:|
|
||||
| Denver |
|
||||
| Nairobi |
|
||||
| Tokyo |
|
||||
|
||||
You can apply *ordinal encoding* to substitute a unique integer value for each category, like this:
|
||||
|
||||
| city |
|
||||
|:----:|
|
||||
| 0 |
|
||||
| 1 |
|
||||
| 2 |
|
||||
|
||||
And that's what we'll do to our data!
|
||||
|
||||
In this section, we'll explore another amazing Tidymodels package: [recipes](https://tidymodels.github.io/recipes/) - which is designed to help you preprocess your data **before** training your model. At its core, a recipe is an object that defines what steps should be applied to a data set in order to get it ready for modelling.
|
||||
|
||||
Now, let's create a recipe that prepares our data for modelling by substituting a unique integer for all the observations in the predictor columns:
|
||||
|
||||
```{r pumpkins_recipe}
|
||||
# Specify a recipe
|
||||
pumpkins_recipe <- recipe(price ~ ., data = new_pumpkins) %>%
|
||||
step_integer(all_predictors(), zero_based = TRUE)
|
||||
|
||||
|
||||
# Print out the recipe
|
||||
pumpkins_recipe
|
||||
|
||||
```
|
||||
|
||||
Awesome! 👏 We just created our first recipe that specifies an outcome (price) and its corresponding predictors and that all the predictor columns should be encoded into a set of integers 🙌! Let's quickly break it down:
|
||||
|
||||
- The call to `recipe()` with a formula tells the recipe the *roles* of the variables using `new_pumpkins` data as the reference. For instance the `price` column has been assigned an `outcome` role while the rest of the columns have been assigned a `predictor` role.
|
||||
|
||||
- `step_integer(all_predictors(), zero_based = TRUE)` specifies that all the predictors should be converted into a set of integers with the numbering starting at 0.
|
||||
|
||||
We are sure you may be having thoughts such as: "This is so cool!! But what if I needed to confirm that the recipes are doing exactly what I expect them to do? 🤔"
|
||||
|
||||
That's an awesome thought! You see, once your recipe is defined, you can estimate the parameters required to actually preprocess the data, and then extract the processed data. You don't typically need to do this when you use Tidymodels (we'll see the normal convention in just a minute-\> `workflows`) but it can come in handy when you want to do some kind of sanity check for confirming that recipes are doing what you expect.
|
||||
|
||||
For that, you'll need two more verbs: `prep()` and `bake()` and as always, our little R friends by [`Allison Horst`](https://github.com/allisonhorst/stats-illustrations) help you in understanding this better!
|
||||
|
||||
{width="550"}
|
||||
|
||||
[`prep()`](https://recipes.tidymodels.org/reference/prep.html): estimates the required parameters from a training set that can be later applied to other data sets. For instance, for a given predictor column, what observation will be assigned integer 0 or 1 or 2 etc
|
||||
|
||||
[`bake()`](https://recipes.tidymodels.org/reference/bake.html): takes a prepped recipe and applies the operations to any data set.
|
||||
|
||||
That said, lets prep and bake our recipes to really confirm that under the hood, the predictor columns will be first encoded before a model is fit.
|
||||
|
||||
```{r prep_bake}
|
||||
# Prep the recipe
|
||||
pumpkins_prep <- prep(pumpkins_recipe)
|
||||
|
||||
# Bake the recipe to extract a preprocessed new_pumpkins data
|
||||
baked_pumpkins <- bake(pumpkins_prep, new_data = NULL)
|
||||
|
||||
# Print out the baked data set
|
||||
baked_pumpkins %>%
|
||||
slice_head(n = 10)
|
||||
```
|
||||
|
||||
Woo-hoo!🥳 The processed data `baked_pumpkins` has all it's predictors encoded confirming that indeed the preprocessing steps defined as our recipe will work as expected. This makes it harder for you to read but much more intelligible for Tidymodels! Take some time to find out what observation has been mapped to a corresponding integer.
|
||||
|
||||
It is also worth mentioning that `baked_pumpkins` is a data frame that we can perform computations on.
|
||||
|
||||
For instance, let's try to find a good correlation between two points of your data to potentially build a good predictive model. We'll use the function `cor()` to do this. Type `?cor()` to find out more about the function.
|
||||
|
||||
```{r corr}
|
||||
# Find the correlation between the city_name and the price
|
||||
cor(baked_pumpkins$city_name, baked_pumpkins$price)
|
||||
|
||||
# Find the correlation between the package and the price
|
||||
cor(baked_pumpkins$package, baked_pumpkins$price)
|
||||
|
||||
```
|
||||
|
||||
As it turns out, there's only weak correlation between the City and Price. However there's a bit better correlation between the Package and its Price. That makes sense, right? Normally, the bigger the produce box, the higher the price.
|
||||
|
||||
While we are at it, let's also try and visualize a correlation matrix of all the columns using the `corrplot` package.
|
||||
|
||||
```{r corrplot}
|
||||
# Load the corrplot package
|
||||
library(corrplot)
|
||||
|
||||
# Obtain correlation matrix
|
||||
corr_mat <- cor(baked_pumpkins %>%
|
||||
# Drop columns that are not really informative
|
||||
select(-c(low_price, high_price)))
|
||||
|
||||
# Make a correlation plot between the variables
|
||||
corrplot(corr_mat, method = "shade", shade.col = NA, tl.col = "black", tl.srt = 45, addCoef.col = "black", cl.pos = "n", order = "original")
|
||||
|
||||
```
|
||||
|
||||
🤩🤩 Much better.
|
||||
|
||||
A good question to now ask of this data will be: '`What price can I expect of a given pumpkin package?`' Let's get right into it!
|
||||
|
||||
> Note: When you **`bake()`** the prepped recipe **`pumpkins_prep`** with **`new_data = NULL`**, you extract the processed (i.e. encoded) training data. If you had another data set for example a test set and would want to see how a recipe would pre-process it, you would simply bake **`pumpkins_prep`** with **`new_data = test_set`**
|
||||
|
||||
## 4. Build a linear regression model
|
||||
|
||||
{width="800"}
|
||||
|
||||
Now that we have build a recipe, and actually confirmed that the data will be pre-processed appropriately, let's now build a regression model to answer the question: `What price can I expect of a given pumpkin package?`
|
||||
|
||||
#### Train a linear regression model using the training set
|
||||
|
||||
As you may have already figured out, the column *price* is the `outcome` variable while the *package* column is the `predictor` variable.
|
||||
|
||||
To do this, we'll first split the data such that 80% goes into training and 20% into test set, then define a recipe that will encode the predictor column into a set of integers, then build a model specification. We won't prep and bake our recipe since we already know it will preprocess the data as expected.
|
||||
|
||||
```{r lm_rec_spec}
|
||||
set.seed(2056)
|
||||
# Split the data into training and test sets
|
||||
pumpkins_split <- new_pumpkins %>%
|
||||
initial_split(prop = 0.8)
|
||||
|
||||
|
||||
# Extract training and test data
|
||||
pumpkins_train <- training(pumpkins_split)
|
||||
pumpkins_test <- testing(pumpkins_split)
|
||||
|
||||
|
||||
|
||||
# Create a recipe for preprocessing the data
|
||||
lm_pumpkins_recipe <- recipe(price ~ package, data = pumpkins_train) %>%
|
||||
step_integer(all_predictors(), zero_based = TRUE)
|
||||
|
||||
|
||||
|
||||
# Create a linear model specification
|
||||
lm_spec <- linear_reg() %>%
|
||||
set_engine("lm") %>%
|
||||
set_mode("regression")
|
||||
|
||||
|
||||
```
|
||||
|
||||
Good job! Now that we have a recipe and a model specification, we need to find a way of bundling them together into an object that will first preprocess the data (prep+bake behind the scenes), fit the model on the preprocessed data and also allow for potential post-processing activities. How's that for your peace of mind!🤩
|
||||
|
||||
In Tidymodels, this convenient object is called a [`workflow`](https://workflows.tidymodels.org/) and conveniently holds your modeling components! This is what we'd call *pipelines* in *Python*.
|
||||
|
||||
So let's bundle everything up into a workflow!📦
|
||||
|
||||
```{r lm_workflow}
|
||||
# Hold modelling components in a workflow
|
||||
lm_wf <- workflow() %>%
|
||||
add_recipe(lm_pumpkins_recipe) %>%
|
||||
add_model(lm_spec)
|
||||
|
||||
# Print out the workflow
|
||||
lm_wf
|
||||
|
||||
```
|
||||
|
||||
👌 Into the bargain, a workflow can be fit/trained in much the same way a model can.
|
||||
|
||||
```{r lm_wf_fit}
|
||||
# Train the model
|
||||
lm_wf_fit <- lm_wf %>%
|
||||
fit(data = pumpkins_train)
|
||||
|
||||
# Print the model coefficients learned
|
||||
lm_wf_fit
|
||||
|
||||
```
|
||||
|
||||
From the model output, we can see the coefficients learned during training. They represent the coefficients of the line of best fit that gives us the lowest overall error between the actual and predicted variable.
|
||||
|
||||
#### Evaluate model performance using the test set
|
||||
|
||||
It's time to see how the model performed 📏! How do we do this?
|
||||
|
||||
Now that we've trained the model, we can use it to make predictions for the test_set using `parsnip::predict()`. Then we can compare these predictions to the actual label values to evaluate how well (or not!) the model is working.
|
||||
|
||||
Let's start with making predictions for the test set then bind the columns to the test set.
|
||||
|
||||
```{r lm_pred}
|
||||
# Make predictions for the test set
|
||||
predictions <- lm_wf_fit %>%
|
||||
predict(new_data = pumpkins_test)
|
||||
|
||||
|
||||
# Bind predictions to the test set
|
||||
lm_results <- pumpkins_test %>%
|
||||
select(c(package, price)) %>%
|
||||
bind_cols(predictions)
|
||||
|
||||
|
||||
# Print the first ten rows of the tibble
|
||||
lm_results %>%
|
||||
slice_head(n = 10)
|
||||
```
|
||||
|
||||
Yes, you have just trained a model and used it to make predictions!🔮 Is it any good, let's evaluate the model's performance!
|
||||
|
||||
In Tidymodels, we do this using `yardstick::metrics()`! For linear regression, let's focus on the following metrics:
|
||||
|
||||
- `Root Mean Square Error (RMSE)`: The square root of the [MSE](https://en.wikipedia.org/wiki/Mean_squared_error). This yields an absolute metric in the same unit as the label (in this case, the price of a pumpkin). The smaller the value, the better the model (in a simplistic sense, it represents the average price by which the predictions are wrong!)
|
||||
|
||||
- `Coefficient of Determination (usually known as R-squared or R2)`: A relative metric in which the higher the value, the better the fit of the model. In essence, this metric represents how much of the variance between predicted and actual label values the model is able to explain.
|
||||
|
||||
```{r lm_yardstick}
|
||||
# Evaluate performance of linear regression
|
||||
metrics(data = lm_results,
|
||||
truth = price,
|
||||
estimate = .pred)
|
||||
|
||||
|
||||
```
|
||||
|
||||
There goes the model performance. Let's see if we can get a better indication by visualizing a scatter plot of the package and price then use the predictions made to overlay a line of best fit.
|
||||
|
||||
This means we'll have to prep and bake the test set in order to encode the package column then bind this to the predictions made by our model.
|
||||
|
||||
```{r lm_plot}
|
||||
# Encode package column
|
||||
package_encode <- lm_pumpkins_recipe %>%
|
||||
prep() %>%
|
||||
bake(new_data = pumpkins_test) %>%
|
||||
select(package)
|
||||
|
||||
|
||||
# Bind encoded package column to the results
|
||||
lm_results <- lm_results %>%
|
||||
bind_cols(package_encode %>%
|
||||
rename(package_integer = package)) %>%
|
||||
relocate(package_integer, .after = package)
|
||||
|
||||
|
||||
# Print new results data frame
|
||||
lm_results %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
|
||||
# Make a scatter plot
|
||||
lm_results %>%
|
||||
ggplot(mapping = aes(x = package_integer, y = price)) +
|
||||
geom_point(size = 1.6) +
|
||||
# Overlay a line of best fit
|
||||
geom_line(aes(y = .pred), color = "orange", size = 1.2) +
|
||||
xlab("package")
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
Great! As you can see, the linear regression model does not really well generalize the relationship between a package and its corresponding price.
|
||||
|
||||
🎃 Congratulations, you just created a model that can help predict the price of a few varieties of pumpkins. Your holiday pumpkin patch will be beautiful. But you can probably create a better model!
|
||||
|
||||
## 5. Build a polynomial regression model
|
||||
|
||||
{width="800"}
|
||||
|
||||
Sometimes our data may not have a linear relationship, but we still want to predict an outcome. Polynomial regression can help us make predictions for more complex non-linear relationships.
|
||||
|
||||
Take for instance the relationship between the package and price for our pumpkins data set. While sometimes there's a linear relationship between variables - the bigger the pumpkin in volume, the higher the price - sometimes these relationships can't be plotted as a plane or straight line.
|
||||
|
||||
> ✅ Here are [some more examples](https://online.stat.psu.edu/stat501/lesson/9/9.8) of data that could use polynomial regression
|
||||
>
|
||||
> Take another look at the relationship between Variety to Price in the previous plot. Does this scatterplot seem like it should necessarily be analyzed by a straight line? Perhaps not. In this case, you can try polynomial regression.
|
||||
>
|
||||
> ✅ Polynomials are mathematical expressions that might consist of one or more variables and coefficients
|
||||
|
||||
#### Train a polynomial regression model using the training set
|
||||
|
||||
Polynomial regression creates a *curved line* to better fit nonlinear data.
|
||||
|
||||
Let's see whether a polynomial model will perform better in making predictions. We'll follow a somewhat similar procedure as we did before:
|
||||
|
||||
- Create a recipe that specifies the preprocessing steps that should be carried out on our data to get it ready for modelling i.e: encoding predictors and computing polynomials of degree *n*
|
||||
|
||||
- Build a model specification
|
||||
|
||||
- Bundle the recipe and model specification into a workflow
|
||||
|
||||
- Create a model by fitting the workflow
|
||||
|
||||
- Evaluate how well the model performs on the test data
|
||||
|
||||
Let's get right into it!
|
||||
|
||||
```{r polynomial_reg}
|
||||
# Specify a recipe
|
||||
poly_pumpkins_recipe <-
|
||||
recipe(price ~ package, data = pumpkins_train) %>%
|
||||
step_integer(all_predictors(), zero_based = TRUE) %>%
|
||||
step_poly(all_predictors(), degree = 4)
|
||||
|
||||
|
||||
# Create a model specification
|
||||
poly_spec <- linear_reg() %>%
|
||||
set_engine("lm") %>%
|
||||
set_mode("regression")
|
||||
|
||||
|
||||
# Bundle recipe and model spec into a workflow
|
||||
poly_wf <- workflow() %>%
|
||||
add_recipe(poly_pumpkins_recipe) %>%
|
||||
add_model(poly_spec)
|
||||
|
||||
|
||||
# Create a model
|
||||
poly_wf_fit <- poly_wf %>%
|
||||
fit(data = pumpkins_train)
|
||||
|
||||
|
||||
# Print learned model coefficients
|
||||
poly_wf_fit
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
#### Evaluate model performance
|
||||
|
||||
👏👏You've built a polynomial model let's make predictions on the test set!
|
||||
|
||||
```{r poly_predict}
|
||||
# Make price predictions on test data
|
||||
poly_results <- poly_wf_fit %>% predict(new_data = pumpkins_test) %>%
|
||||
bind_cols(pumpkins_test %>% select(c(package, price))) %>%
|
||||
relocate(.pred, .after = last_col())
|
||||
|
||||
|
||||
# Print the results
|
||||
poly_results %>%
|
||||
slice_head(n = 10)
|
||||
```
|
||||
|
||||
Woo-hoo , let's evaluate how the model performed on the test_set using `yardstick::metrics()`.
|
||||
|
||||
```{r poly_eval}
|
||||
metrics(data = poly_results, truth = price, estimate = .pred)
|
||||
```
|
||||
|
||||
🤩🤩 Much better performance.
|
||||
|
||||
The `rmse` decreased from about 7. to about 3. an indication that of a reduced error between the actual price and the predicted price. You can *loosely* interpret this as meaning that on average, incorrect predictions are wrong by around \$3. The `rsq` increased from about 0.4 to 0.8.
|
||||
|
||||
All these metrics indicate that the polynomial model performs way better than the linear model. Good job!
|
||||
|
||||
Let's see if we can visualize this!
|
||||
|
||||
```{r poly_viz}
|
||||
# Bind encoded package column to the results
|
||||
poly_results <- poly_results %>%
|
||||
bind_cols(package_encode %>%
|
||||
rename(package_integer = package)) %>%
|
||||
relocate(package_integer, .after = package)
|
||||
|
||||
|
||||
# Print new results data frame
|
||||
poly_results %>%
|
||||
slice_head(n = 5)
|
||||
|
||||
|
||||
# Make a scatter plot
|
||||
poly_results %>%
|
||||
ggplot(mapping = aes(x = package_integer, y = price)) +
|
||||
geom_point(size = 1.6) +
|
||||
# Overlay a line of best fit
|
||||
geom_line(aes(y = .pred), color = "midnightblue", size = 1.2) +
|
||||
xlab("package")
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
You can see a curved line that fits your data better! 🤩
|
||||
|
||||
You can make this more smoother by passing a polynomial formula to `geom_smooth` like this:
|
||||
|
||||
```{r smooth curve}
|
||||
# Make a scatter plot
|
||||
poly_results %>%
|
||||
ggplot(mapping = aes(x = package_integer, y = price)) +
|
||||
geom_point(size = 1.6) +
|
||||
# Overlay a line of best fit
|
||||
geom_smooth(method = lm, formula = y ~ poly(x, degree = 4), color = "midnightblue", size = 1.2, se = FALSE) +
|
||||
xlab("package")
|
||||
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
Much like a smooth curve!🤩
|
||||
|
||||
Here's how you would make a new prediction:
|
||||
|
||||
```{r predict}
|
||||
# Make a hypothetical data frame
|
||||
hypo_tibble <- tibble(package = "bushel baskets")
|
||||
|
||||
# Make predictions using linear model
|
||||
lm_pred <- lm_wf_fit %>% predict(new_data = hypo_tibble)
|
||||
|
||||
# Make predictions using polynomial model
|
||||
poly_pred <- poly_wf_fit %>% predict(new_data = hypo_tibble)
|
||||
|
||||
# Return predictions in a list
|
||||
list("linear model prediction" = lm_pred,
|
||||
"polynomial model prediction" = poly_pred)
|
||||
|
||||
|
||||
```
|
||||
|
||||
The `polynomial model` prediction does make sense, given the scatter plots of `price` and `package`! And, if this is a better model than the previous one, looking at the same data, you need to budget for these more expensive pumpkins!
|
||||
|
||||
🏆 Well done! You created two regression models in one lesson. In the final section on regression, you will learn about logistic regression to determine categories.
|
||||
|
||||
## **🚀Challenge**
|
||||
|
||||
Test several different variables in this notebook to see how correlation corresponds to model accuracy.
|
||||
|
||||
## [**Post-lecture quiz**](https://gray-sand-07a10f403.1.azurestaticapps.net/quiz/14/)
|
||||
|
||||
## **Review & Self Study**
|
||||
|
||||
In this lesson we learned about Linear Regression. There are other important types of Regression. Read about Stepwise, Ridge, Lasso and Elasticnet techniques. A good course to study to learn more is the [Stanford Statistical Learning course](https://online.stanford.edu/courses/sohs-ystatslearning-statistical-learning)
|
||||
|
||||
If you want to learn more about how to use the amazing Tidymodels framework, please check out the following resources:
|
||||
|
||||
- Tidymodels website: [Get started with Tidymodels](https://www.tidymodels.org/start/)
|
||||
|
||||
- Max Kuhn and Julia Silge, [*Tidy Modeling with R*](https://www.tmwr.org/)*.*
|
||||
|
||||
###### **THANK YOU TO:**
|
||||
|
||||
[Allison Horst](https://twitter.com/allison_horst?lang=en) for creating the amazing illustrations that make R more welcoming and engaging. Find more illustrations at her [gallery](https://www.google.com/url?q=https://github.com/allisonhorst/stats-illustrations&sa=D&source=editors&ust=1626380772530000&usg=AOvVaw3zcfyCizFQZpkSLzxiiQEM).
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,395 @@
|
||||
# Logistic regression to predict categories
|
||||
|
||||
|
||||
|
||||
## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
> ### [This lesson is available in R!](./solution/R/lesson_4.html)
|
||||
|
||||
## Introduction
|
||||
|
||||
In this final lesson on Regression, one of the basic _classic_ ML techniques, we will take a look at Logistic Regression. You would use this technique to discover patterns to predict binary categories. Is this candy chocolate or not? Is this disease contagious or not? Will this customer choose this product or not?
|
||||
|
||||
In this lesson, you will learn:
|
||||
|
||||
- A new library for data visualization
|
||||
- Techniques for logistic regression
|
||||
|
||||
✅ Deepen your understanding of working with this type of regression in this [Learn module](https://docs.microsoft.com/learn/modules/train-evaluate-classification-models?WT.mc_id=academic-77952-leestott)
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Having worked with the pumpkin data, we are now familiar enough with it to realize that there's one binary category that we can work with: `Color`.
|
||||
|
||||
Let's build a logistic regression model to predict that, given some variables, _what color a given pumpkin is likely to be_ (orange 🎃 or white 👻).
|
||||
|
||||
> Why are we talking about binary classification in a lesson grouping about regression? Only for linguistic convenience, as logistic regression is [really a classification method](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression), albeit a linear-based one. Learn about other ways to classify data in the next lesson group.
|
||||
|
||||
## Define the question
|
||||
|
||||
For our purposes, we will express this as a binary: 'White' or 'Not White'. There is also a 'striped' category in our dataset but there are few instances of it, so we will not use it. It disappears once we remove null values from the dataset, anyway.
|
||||
|
||||
> 🎃 Fun fact, we sometimes call white pumpkins 'ghost' pumpkins. They aren't very easy to carve, so they aren't as popular as the orange ones but they are cool looking! So we could also reformulate our question as: 'Ghost' or 'Not Ghost'. 👻
|
||||
|
||||
## About logistic regression
|
||||
|
||||
Logistic regression differs from linear regression, which you learned about previously, in a few important ways.
|
||||
|
||||
[](https://youtu.be/KpeCT6nEpBY "ML for beginners - Understanding Logistic Regression for Machine Learning Classification")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of logistic regression.
|
||||
|
||||
### Binary classification
|
||||
|
||||
Logistic regression does not offer the same features as linear regression. The former offers a prediction about a binary category ("white or not white") whereas the latter is capable of predicting continual values, for example given the origin of a pumpkin and the time of harvest, _how much its price will rise_.
|
||||
|
||||
|
||||
> Infographic by [Dasani Madipalli](https://twitter.com/dasani_decoded)
|
||||
|
||||
### Other classifications
|
||||
|
||||
There are other types of logistic regression, including multinomial and ordinal:
|
||||
|
||||
- **Multinomial**, which involves having more than one category - "Orange, White, and Striped".
|
||||
- **Ordinal**, which involves ordered categories, useful if we wanted to order our outcomes logically, like our pumpkins that are ordered by a finite number of sizes (mini,sm,med,lg,xl,xxl).
|
||||
|
||||
|
||||
|
||||
### Variables DO NOT have to correlate
|
||||
|
||||
Remember how linear regression worked better with more correlated variables? Logistic regression is the opposite - the variables don't have to align. That works for this data which has somewhat weak correlations.
|
||||
|
||||
### You need a lot of clean data
|
||||
|
||||
Logistic regression will give more accurate results if you use more data; our small dataset is not optimal for this task, so keep that in mind.
|
||||
|
||||
[](https://youtu.be/B2X4H9vcXTs "ML for beginners - Data Analysis and Preparation for Logistic Regression")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of preparing data for linear regression
|
||||
|
||||
✅ Think about the types of data that would lend themselves well to logistic regression
|
||||
|
||||
## Exercise - tidy the data
|
||||
|
||||
First, clean the data a bit, dropping null values and selecting only some of the columns:
|
||||
|
||||
1. Add the following code:
|
||||
|
||||
```python
|
||||
|
||||
columns_to_select = ['City Name','Package','Variety', 'Origin','Item Size', 'Color']
|
||||
pumpkins = full_pumpkins.loc[:, columns_to_select]
|
||||
|
||||
pumpkins.dropna(inplace=True)
|
||||
```
|
||||
|
||||
You can always take a peek at your new dataframe:
|
||||
|
||||
```python
|
||||
pumpkins.info
|
||||
```
|
||||
|
||||
### Visualization - categorical plot
|
||||
|
||||
By now you have loaded up the [starter notebook](./notebook.ipynb) with pumpkin data once again and cleaned it so as to preserve a dataset containing a few variables, including `Color`. Let's visualize the dataframe in the notebook using a different library: [Seaborn](https://seaborn.pydata.org/index.html), which is built on Matplotlib which we used earlier.
|
||||
|
||||
Seaborn offers some neat ways to visualize your data. For example, you can compare distributions of the data for each `Variety` and `Color` in a categorical plot.
|
||||
|
||||
1. Create such a plot by using the `catplot` function, using our pumpkin data `pumpkins`, and specifying a color mapping for each pumpkin category (orange or white):
|
||||
|
||||
```python
|
||||
import seaborn as sns
|
||||
|
||||
palette = {
|
||||
'ORANGE': 'orange',
|
||||
'WHITE': 'wheat',
|
||||
}
|
||||
|
||||
sns.catplot(
|
||||
data=pumpkins, y="Variety", hue="Color", kind="count",
|
||||
palette=palette,
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
By observing the data, you can see how the Color data relates to Variety.
|
||||
|
||||
✅ Given this categorical plot, what are some interesting explorations you can envision?
|
||||
|
||||
### Data pre-processing: feature and label encoding
|
||||
Our pumpkins dataset contains string values for all its columns. Working with categorical data is intuitive for humans but not for machines. Machine learning algorithms work well with numbers. That's why encoding is a very important step in the data pre-processing phase, since it enables us to turn categorical data into numerical data, without losing any information. Good encoding leads to building a good model.
|
||||
|
||||
For feature encoding there are two main types of encoders:
|
||||
|
||||
1. Ordinal encoder: it suits well for ordinal variables, which are categorical variables where their data follows a logical ordering, like the `Item Size` column in our dataset. It creates a mapping such that each category is represented by a number, which is the order of the category in the column.
|
||||
|
||||
```python
|
||||
from sklearn.preprocessing import OrdinalEncoder
|
||||
|
||||
item_size_categories = [['sml', 'med', 'med-lge', 'lge', 'xlge', 'jbo', 'exjbo']]
|
||||
ordinal_features = ['Item Size']
|
||||
ordinal_encoder = OrdinalEncoder(categories=item_size_categories)
|
||||
```
|
||||
|
||||
2. Categorical encoder: it suits well for nominal variables, which are categorical variables where their data does not follow a logical ordering, like all the features different from `Item Size` in our dataset. It is a one-hot encoding, which means that each category is represented by a binary column: the encoded variable is equal to 1 if the pumpkin belongs to that Variety and 0 otherwise.
|
||||
|
||||
```python
|
||||
from sklearn.preprocessing import OneHotEncoder
|
||||
|
||||
categorical_features = ['City Name', 'Package', 'Variety', 'Origin']
|
||||
categorical_encoder = OneHotEncoder(sparse_output=False)
|
||||
```
|
||||
Then, `ColumnTransformer` is used to combine multiple encoders into a single step and apply them to the appropriate columns.
|
||||
|
||||
```python
|
||||
from sklearn.compose import ColumnTransformer
|
||||
|
||||
ct = ColumnTransformer(transformers=[
|
||||
('ord', ordinal_encoder, ordinal_features),
|
||||
('cat', categorical_encoder, categorical_features)
|
||||
])
|
||||
|
||||
ct.set_output(transform='pandas')
|
||||
encoded_features = ct.fit_transform(pumpkins)
|
||||
```
|
||||
On the other hand, to encode the label, we use the scikit-learn `LabelEncoder` class, which is a utility class to help normalize labels such that they contain only values between 0 and n_classes-1 (here, 0 and 1).
|
||||
|
||||
```python
|
||||
from sklearn.preprocessing import LabelEncoder
|
||||
|
||||
label_encoder = LabelEncoder()
|
||||
encoded_label = label_encoder.fit_transform(pumpkins['Color'])
|
||||
```
|
||||
Once we have encoded the features and the label, we can merge them into a new dataframe `encoded_pumpkins`.
|
||||
|
||||
```python
|
||||
encoded_pumpkins = encoded_features.assign(Color=encoded_label)
|
||||
```
|
||||
✅ What are the advantages of using an ordinal encoder for the `Item Size` column?
|
||||
|
||||
### Analyse relationships between variables
|
||||
|
||||
Now that we have pre-processed our data, we can analyse the relationships between the features and the label to grasp an idea of how well the model will be able to predict the label given the features.
|
||||
The best way to perform this kind of analysis is plotting the data. We'll be using again the Seaborn `catplot` function, to visualize the relationships between `Item Size`, `Variety` and `Color` in a categorical plot. To better plot the data we'll be using the encoded `Item Size` column and the unencoded `Variety` column.
|
||||
|
||||
```python
|
||||
palette = {
|
||||
'ORANGE': 'orange',
|
||||
'WHITE': 'wheat',
|
||||
}
|
||||
pumpkins['Item Size'] = encoded_pumpkins['ord__Item Size']
|
||||
|
||||
g = sns.catplot(
|
||||
data=pumpkins,
|
||||
x="Item Size", y="Color", row='Variety',
|
||||
kind="box", orient="h",
|
||||
sharex=False, margin_titles=True,
|
||||
height=1.8, aspect=4, palette=palette,
|
||||
)
|
||||
g.set(xlabel="Item Size", ylabel="").set(xlim=(0,6))
|
||||
g.set_titles(row_template="{row_name}")
|
||||
```
|
||||
|
||||
|
||||
### Use a swarm plot
|
||||
|
||||
Since Color is a binary category (White or Not), it needs 'a [specialized approach](https://seaborn.pydata.org/tutorial/categorical.html?highlight=bar) to visualization'. There are other ways to visualize the relationship of this category with other variables.
|
||||
|
||||
You can visualize variables side-by-side with Seaborn plots.
|
||||
|
||||
1. Try a 'swarm' plot to show the distribution of values:
|
||||
|
||||
```python
|
||||
palette = {
|
||||
0: 'orange',
|
||||
1: 'wheat'
|
||||
}
|
||||
sns.swarmplot(x="Color", y="ord__Item Size", data=encoded_pumpkins, palette=palette)
|
||||
```
|
||||
|
||||
|
||||
|
||||
**Watch Out**: the code above might generate a warning, since seaborn fails to represent such amount of datapoints into a swam plot. A possible solution is decreasing the size of the marker, by using the 'size' parameter. However, be aware that this affects the readability of the plot.
|
||||
|
||||
|
||||
> **🧮 Show Me The Math**
|
||||
>
|
||||
> Logistic regression relies on the concept of 'maximum likelihood' using [sigmoid functions](https://wikipedia.org/wiki/Sigmoid_function). A 'Sigmoid Function' on a plot looks like an 'S' shape. It takes a value and maps it to somewhere between 0 and 1. Its curve is also called a 'logistic curve'. Its formula looks like this:
|
||||
>
|
||||
> 
|
||||
>
|
||||
> where the sigmoid's midpoint finds itself at x's 0 point, L is the curve's maximum value, and k is the curve's steepness. If the outcome of the function is more than 0.5, the label in question will be given the class '1' of the binary choice. If not, it will be classified as '0'.
|
||||
|
||||
## Build your model
|
||||
|
||||
Building a model to find these binary classification is surprisingly straightforward in Scikit-learn.
|
||||
|
||||
[](https://youtu.be/MmZS2otPrQ8 "ML for beginners - Logistic Regression for classification of data")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of building a linear regression model
|
||||
|
||||
1. Select the variables you want to use in your classification model and split the training and test sets calling `train_test_split()`:
|
||||
|
||||
```python
|
||||
from sklearn.model_selection import train_test_split
|
||||
|
||||
X = encoded_pumpkins[encoded_pumpkins.columns.difference(['Color'])]
|
||||
y = encoded_pumpkins['Color']
|
||||
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
|
||||
|
||||
```
|
||||
|
||||
2. Now you can train your model, by calling `fit()` with your training data, and print out its result:
|
||||
|
||||
```python
|
||||
from sklearn.metrics import f1_score, classification_report
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
|
||||
model = LogisticRegression()
|
||||
model.fit(X_train, y_train)
|
||||
predictions = model.predict(X_test)
|
||||
|
||||
print(classification_report(y_test, predictions))
|
||||
print('Predicted labels: ', predictions)
|
||||
print('F1-score: ', f1_score(y_test, predictions))
|
||||
```
|
||||
|
||||
Take a look at your model's scoreboard. It's not bad, considering you have only about 1000 rows of data:
|
||||
|
||||
```output
|
||||
precision recall f1-score support
|
||||
|
||||
0 0.94 0.98 0.96 166
|
||||
1 0.85 0.67 0.75 33
|
||||
|
||||
accuracy 0.92 199
|
||||
macro avg 0.89 0.82 0.85 199
|
||||
weighted avg 0.92 0.92 0.92 199
|
||||
|
||||
Predicted labels: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0
|
||||
0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
|
||||
1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0
|
||||
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0
|
||||
0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
|
||||
0 0 0 1 0 0 0 0 0 0 0 0 1 1]
|
||||
F1-score: 0.7457627118644068
|
||||
```
|
||||
|
||||
## Better comprehension via a confusion matrix
|
||||
|
||||
While you can get a scoreboard report [terms](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html?highlight=classification_report#sklearn.metrics.classification_report) by printing out the items above, you might be able to understand your model more easily by using a [confusion matrix](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix) to help us understand how the model is performing.
|
||||
|
||||
> 🎓 A '[confusion matrix](https://wikipedia.org/wiki/Confusion_matrix)' (or 'error matrix') is a table that expresses your model's true vs. false positives and negatives, thus gauging the accuracy of predictions.
|
||||
|
||||
1. To use a confusion metrics, call `confusion_matrix()`:
|
||||
|
||||
```python
|
||||
from sklearn.metrics import confusion_matrix
|
||||
confusion_matrix(y_test, predictions)
|
||||
```
|
||||
|
||||
Take a look at your model's confusion matrix:
|
||||
|
||||
```output
|
||||
array([[162, 4],
|
||||
[ 11, 22]])
|
||||
```
|
||||
|
||||
In Scikit-learn, confusion matrices Rows (axis 0) are actual labels and columns (axis 1) are predicted labels.
|
||||
|
||||
| | 0 | 1 |
|
||||
| :---: | :---: | :---: |
|
||||
| 0 | TN | FP |
|
||||
| 1 | FN | TP |
|
||||
|
||||
What's going on here? Let's say our model is asked to classify pumpkins between two binary categories, category 'white' and category 'not-white'.
|
||||
|
||||
- If your model predicts a pumpkin as not white and it belongs to category 'not-white' in reality we call it a true negative, shown by the top left number.
|
||||
- If your model predicts a pumpkin as white and it belongs to category 'not-white' in reality we call it a false negative, shown by the bottom left number.
|
||||
- If your model predicts a pumpkin as not white and it belongs to category 'white' in reality we call it a false positive, shown by the top right number.
|
||||
- If your model predicts a pumpkin as white and it belongs to category 'white' in reality we call it a true positive, shown by the bottom right number.
|
||||
|
||||
As you might have guessed it's preferable to have a larger number of true positives and true negatives and a lower number of false positives and false negatives, which implies that the model performs better.
|
||||
|
||||
How does the confusion matrix relate to precision and recall? Remember, the classification report printed above showed precision (0.85) and recall (0.67).
|
||||
|
||||
Precision = tp / (tp + fp) = 22 / (22 + 4) = 0.8461538461538461
|
||||
|
||||
Recall = tp / (tp + fn) = 22 / (22 + 11) = 0.6666666666666666
|
||||
|
||||
✅ Q: According to the confusion matrix, how did the model do? A: Not bad; there are a good number of true negatives but also a few false negatives.
|
||||
|
||||
Let's revisit the terms we saw earlier with the help of the confusion matrix's mapping of TP/TN and FP/FN:
|
||||
|
||||
🎓 Precision: TP/(TP + FP) The fraction of relevant instances among the retrieved instances (e.g. which labels were well-labeled)
|
||||
|
||||
🎓 Recall: TP/(TP + FN) The fraction of relevant instances that were retrieved, whether well-labeled or not
|
||||
|
||||
🎓 f1-score: (2 * precision * recall)/(precision + recall) A weighted average of the precision and recall, with best being 1 and worst being 0
|
||||
|
||||
🎓 Support: The number of occurrences of each label retrieved
|
||||
|
||||
🎓 Accuracy: (TP + TN)/(TP + TN + FP + FN) The percentage of labels predicted accurately for a sample.
|
||||
|
||||
🎓 Macro Avg: The calculation of the unweighted mean metrics for each label, not taking label imbalance into account.
|
||||
|
||||
🎓 Weighted Avg: The calculation of the mean metrics for each label, taking label imbalance into account by weighting them by their support (the number of true instances for each label).
|
||||
|
||||
✅ Can you think which metric you should watch if you want your model to reduce the number of false negatives?
|
||||
|
||||
## Visualize the ROC curve of this model
|
||||
|
||||
[](https://youtu.be/GApO575jTA0 "ML for beginners - Analyzing Logistic Regression Performance with ROC Curves")
|
||||
|
||||
> 🎥 Click the image above for a short video overview of ROC curves
|
||||
|
||||
Let's do one more visualization to see the so-called 'ROC' curve:
|
||||
|
||||
```python
|
||||
from sklearn.metrics import roc_curve, roc_auc_score
|
||||
import matplotlib
|
||||
import matplotlib.pyplot as plt
|
||||
%matplotlib inline
|
||||
|
||||
y_scores = model.predict_proba(X_test)
|
||||
fpr, tpr, thresholds = roc_curve(y_test, y_scores[:,1])
|
||||
|
||||
fig = plt.figure(figsize=(6, 6))
|
||||
plt.plot([0, 1], [0, 1], 'k--')
|
||||
plt.plot(fpr, tpr)
|
||||
plt.xlabel('False Positive Rate')
|
||||
plt.ylabel('True Positive Rate')
|
||||
plt.title('ROC Curve')
|
||||
plt.show()
|
||||
```
|
||||
|
||||
Using Matplotlib, plot the model's [Receiving Operating Characteristic](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html?highlight=roc) or ROC. ROC curves are often used to get a view of the output of a classifier in terms of its true vs. false positives. "ROC curves typically feature true positive rate on the Y axis, and false positive rate on the X axis." Thus, the steepness of the curve and the space between the midpoint line and the curve matter: you want a curve that quickly heads up and over the line. In our case, there are false positives to start with, and then the line heads up and over properly:
|
||||
|
||||
|
||||
|
||||
Finally, use Scikit-learn's [`roc_auc_score` API](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html?highlight=roc_auc#sklearn.metrics.roc_auc_score) to compute the actual 'Area Under the Curve' (AUC):
|
||||
|
||||
```python
|
||||
auc = roc_auc_score(y_test,y_scores[:,1])
|
||||
print(auc)
|
||||
```
|
||||
The result is `0.9749908725812341`. Given that the AUC ranges from 0 to 1, you want a big score, since a model that is 100% correct in its predictions will have an AUC of 1; in this case, the model is _pretty good_.
|
||||
|
||||
In future lessons on classifications, you will learn how to iterate to improve your model's scores. But for now, congratulations! You've completed these regression lessons!
|
||||
|
||||
---
|
||||
## 🚀Challenge
|
||||
|
||||
There's a lot more to unpack regarding logistic regression! But the best way to learn is to experiment. Find a dataset that lends itself to this type of analysis and build a model with it. What do you learn? tip: try [Kaggle](https://www.kaggle.com/search?q=logistic+regression+datasets) for interesting datasets.
|
||||
|
||||
## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ml/)
|
||||
|
||||
## Review & Self Study
|
||||
|
||||
Read the first few pages of [this paper from Stanford](https://web.stanford.edu/~jurafsky/slp3/5.pdf) on some practical uses for logistic regression. Think about tasks that are better suited for one or the other type of regression tasks that we have studied up to this point. What would work best?
|
||||
|
||||
## Assignment
|
||||
|
||||
[Retrying this regression](assignment.md)
|
||||
@@ -0,0 +1,10 @@
|
||||
# Retrying some Regression
|
||||
|
||||
## Instructions
|
||||
|
||||
In the lesson, you used a subset of the pumpkin data. Now, go back to the original data and try to use all of it, cleaned and standardized, to build a Logistic Regression model.
|
||||
## Rubric
|
||||
|
||||
| Criteria | Exemplary | Adequate | Needs Improvement |
|
||||
| -------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------- |
|
||||
| | A notebook is presented with a well-explained and well-performing model | A notebook is presented with a model that performs minimally | A notebook is presented with a sub-performing model or none |
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 24 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 554 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 185 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 211 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 723 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 176 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 21 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user