chore: import upstream snapshot with attribution

lessons/4-ComputerVision/06-IntroCV/README.md

@@ -0,0 +1,109 @@
+# Introduction to Computer Vision
+
+[Computer Vision](https://wikipedia.org/wiki/Computer_vision) is a discipline whose aim is to allow computers to gain high-level understanding of digital images. This is quite a broad definition, because *understanding* can mean many different things, including finding an object on a picture (**object detection**), understanding what is happening (**event detection**), describing a picture in text, or reconstructing a scene in 3D. There are also special tasks related to human images: age and emotion estimation, face detection and identification, and 3D pose estimation, to name a few.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/11)
+
+One of the simplest tasks of computer vision is **image classification**.
+
+Computer vision is often considered to be a branch of AI. Nowadays, most of computer vision tasks are solved using neural networks. We will learn more about the special type of neural networks used for computer vision, [convolutional neural networks](../07-ConvNets/README.md), throughout this section.
+
+However, before you pass the image to a neural network, in many cases it makes sense to use some algorithmic techniques to enhance the image.
+
+There are several Python libraries available for image processing:
+
+* **[imageio](https://imageio.readthedocs.io/en/stable/)** can be used for reading/writing different image formats. It also support ffmpeg, a useful tool to convert video frames to images.
+* **[Pillow](https://pillow.readthedocs.io/en/stable/index.html)** (also known as PIL) is a bit more powerful, and also supports some image manipulation such as morphing, palette adjustments, and more.
+* **[OpenCV](https://opencv.org/)** is a powerful image processing library written in C++, which has become the *de facto* standard for image processing. It has a convenient Python interface.
+* **[dlib](http://dlib.net/)** is a C++ library that implements many machine learning algorithms, including some of the Computer Vision algorithms. It also has a Python interface, and can be used for challenging tasks such as face and facial landmark detection.
+
+## OpenCV
+
+[OpenCV](https://opencv.org/) is considered to be the *de facto* standard for image processing. It contains a lot of useful algorithms, implemented in C++. You can call OpenCV from Python as well.
+
+A good place to learn OpenCV is [this Learn OpenCV course](https://learnopencv.com/getting-started-with-opencv/). In our curriculum, our goal is not to learn OpenCV, but to show you some examples when it can be used, and how.
+
+### Loading Images
+
+Images in Python can be conveniently represented by NumPy arrays. For example, grayscale images with the size of 320x200 pixels would be stored in a 200x320 array, and color images of the same dimension would have shape of 200x320x3 (for 3 color channels). To load an image, you can use the following code:
+
+```python
+import cv2
+import matplotlib.pyplot as plt
+
+im = cv2.imread('image.jpeg')
+plt.imshow(im)
+```
+
+Traditionally, OpenCV uses BGR (Blue-Green-Red) encoding for color images, while the rest of Python tools use the more traditional RGB (Red-Green-Blue). For the image to look right, you need to convert it to the RGB color space, either by swapping dimensions in the NumPy array, or by calling an OpenCV function:
+
+```python
+im = cv2.cvtColor(im,cv2.COLOR_BGR2RGB)
+```
+
+The same `cvtColor` function can be used to perform other color space transformations such as converting an image to grayscale or to the HSV (Hue-Saturation-Value) color space.
+
+You can also use OpenCV to load video frame-by-frame - an example is given in the exercise [OpenCV Notebook](OpenCV.ipynb).
+
+### Image Processing
+
+Before feeding an image to a neural network, you may want to apply several pre-processing steps. OpenCV can do many things, including:
+
+* **Resizing** the image using `im = cv2.resize(im, (320,200),interpolation=cv2.INTER_LANCZOS)`
+* **Blurring** the image using `im = cv2.medianBlur(im,3)` or `im = cv2.GaussianBlur(im, (3,3), 0)`
+* Changing the **brightness and contrast** of the image can be done by NumPy array manipulations, as described [in this Stackoverflow note](https://stackoverflow.com/questions/39308030/how-do-i-increase-the-contrast-of-an-image-in-python-opencv).
+* Using [thresholding](https://docs.opencv.org/4.x/d7/d4d/tutorial_py_thresholding.html) by calling `cv2.threshold`/`cv2.adaptiveThreshold` functions, which is often preferable to adjusting brightness or contrast.
+* Applying different [transformations](https://docs.opencv.org/4.5.5/da/d6e/tutorial_py_geometric_transformations.html) to the image:
+    - **[Affine transformations](https://docs.opencv.org/4.5.5/d4/d61/tutorial_warp_affine.html)** can be useful if you need to combine rotation, resizing and skewing to the image and you know the source and destination location of three points in the image. Affine transformations keep parallel lines parallel.
+    - **[Perspective transformations](https://medium.com/analytics-vidhya/opencv-perspective-transformation-9edffefb2143)** can be useful when you know the source and destination positions of 4 points in the image. For example, if you take a picture of a rectangular document via a smartphone camera from some angle, and you want to make a rectangular image of the document itself.
+* Understanding movement inside the image by using **[optical flow](https://docs.opencv.org/4.5.5/d4/dee/tutorial_optical_flow.html)**.
+
+## Examples of using Computer Vision
+
+In our [OpenCV Notebook](OpenCV.ipynb), we give some examples of when computer vision can be used to perform specific tasks:
+
+* **Pre-processing a photograph of a Braille book**. We focus on how we can use thresholding, feature detection, perspective transformation and NumPy manipulations to separate individual Braille symbols for further classification by a neural network.
+
+![Braille Image](data/braille.jpeg) | ![Braille Image Pre-processed](images/braille-result.png) | ![Braille Symbols](images/braille-symbols.png)
+----|-----|-----
+
+> Image from [OpenCV.ipynb](OpenCV.ipynb)
+
+* **Detecting motion in video using frame difference**. If the camera is fixed, then frames from the camera feed should be pretty similar to each other. Since frames are represented as arrays, just by subtracting those arrays for two subsequent frames we will get the pixel difference, which should be low for static frames, and become higher once there is substantial motion in the image.
+
+![Image of video frames and frame differences](images/frame-difference.png)
+
+> Image from [OpenCV.ipynb](OpenCV.ipynb)
+
+* **Detecting motion using Optical Flow**. [Optical flow](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html) allows us to understand how individual pixels on video frames move. There are two types of optical flow:
+
+   - **Dense Optical Flow** computes the vector field that shows for each pixel where is it moving
+   - **Sparse Optical Flow** is based on taking some distinctive features in the image (eg. edges), and building their trajectory from frame to frame.
+
+![Image of Optical Flow](images/optical.png)
+
+> Image from [OpenCV.ipynb](OpenCV.ipynb)
+
+## ✍️ Example Notebooks: OpenCV [try OpenCV in Action](OpenCV.ipynb)
+
+Let's do some experiments with OpenCV by exploring [OpenCV Notebook](OpenCV.ipynb)
+
+## Conclusion
+
+Sometimes, relatively complex tasks such as movement detection or fingertip detection can be solved purely by computer vision. Thus, it is very helpful to know the basic techniques of computer vision, and what libraries like OpenCV can do.
+
+## 🚀 Challenge
+
+Watch [this video](https://docs.microsoft.com/shows/ai-show/ai-show--2021-opencv-ai-competition--grand-prize-winners--cortic-tigers--episode-32?WT.mc_id=academic-77998-cacaste) from the AI show to learn about the Cortic Tigers project and how they built a block-based solution to democratize computer vision tasks via a robot. Do some research on other projects like this that help onboard new learners into the field.
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/12)
+
+## Review & Self Study
+
+Read more on optical flow [in this great tutorial](https://learnopencv.com/optical-flow-in-opencv/).
+
+## [Assignment](lab/README.md)
+
+In this lab, you will take a video with simple gestures, and your goal is to extract up/down/left/right movements using optical flow.
+
+<img src="images/palm-movement.png" width="30%" alt="Palm Movement Frame"/>

lessons/4-ComputerVision/06-IntroCV/lab/MovementDetection.ipynb

@@ -0,0 +1,97 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Palm Movement Detection using Optical Flow\n",
+    "\n",
+    "This lab is part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners).\n",
+    "\n",
+    "Consider [this video](palm-movement.mp4), in which a person's palm moves left/right/up/down on the stable background.\n",
+    "\n",
+    "<img src=\"../images/palm-movement.png\" width=\"30%\" alt=\"Palm Movement Frame\"/>\n",
+    "\n",
+    "**Your goal** would be to use Optical Flow to determine, which parts of video contain up/down/left/right movements. \n",
+    "\n",
+    "Start by getting video frames as described in the lecture:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Code here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now, calculate dense optical flow frames as described in the lecture, and convert dense optical flow to polar coordinates: "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Code here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Build histogram of directions for each of the optical flow frame. A histogram shows how many vectors fall under certain bin, and it should separate out different directions of movement on the frame.\n",
+    "\n",
+    "> You may also want to zero out all vectors whose magnitude is below certain threshold. This will get rid of small extra movements in the video, such as eyes and head.\n",
+    "\n",
+    "Plot the histograms for some of the frames."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Code here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Looking at histograms, it should be pretty straightforward how to determine direction of movement. You need so select those bins the correspond to up/down/left/right directions, and that are above certain threshold."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Code here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Congratulations! If you have done all steps above, you have completed the lab!"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

lessons/4-ComputerVision/06-IntroCV/lab/README.md

@@ -0,0 +1,21 @@
+# Detecting Movements using Optical Flow
+
+Lab Assignment from [AI for Beginners Curriculum](https://aka.ms/ai-beginners).
+
+## Task
+
+Consider [this video](palm-movement.mp4), in which a person's palm moves left/right/up/down on the stable background.
+
+<img src="../images/palm-movement.png" width="30%" alt="Palm Movement Frame"/>
+
+**Your goal** would be able to use Optical Flow to determine, which parts of video contain up/down/left/right movements.
+
+**Stretch goal** would be to actually track the palm/finger movement using skin tone, as described [in this blog post](https://dev.to/amarlearning/finger-detection-and-tracking-using-opencv-and-python-586m) or [here](http://www.benmeline.com/finger-tracking-with-opencv-and-python/).
+
+## Starting Notebook
+
+Start the lab by opening [MovementDetection.ipynb](MovementDetection.ipynb)
+
+## Takeaway
+
+Sometimes, relatively complex tasks such as movement detection or fingertip detection can be solved purely by computer vision. Thus, it is very helpful to know what libraries like OpenCV can do.

lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md

@@ -0,0 +1,61 @@
+# Well-Known CNN Architectures
+
+### VGG-16
+
+VGG-16 is a network that achieved 92.7% accuracy in ImageNet top-5 classification in 2014. It has the following layer structure:
+
+![ImageNet Layers](images/vgg-16-arch1.jpg)
+
+As you can see, VGG follows a traditional pyramid architecture, which is a sequence of convolution-pooling layers.
+
+![ImageNet Pyramid](images/vgg-16-arch.jpg)
+
+> Image from [Researchgate](https://www.researchgate.net/figure/Vgg16-model-structure-To-get-the-VGG-NIN-model-we-replace-the-2-nd-4-th-6-th-7-th_fig2_335194493)
+
+### ResNet
+
+ResNet is a family of models proposed by Microsoft Research in 2015. The main idea of ResNet is to use **residual blocks**:
+
+<img src="images/resnet-block.png" width="300"/>
+
+> Image from [this paper](https://arxiv.org/pdf/1512.03385.pdf)
+
+The reason for using identity pass-through is to have our layer predict **the difference** between the result of a previous layer and the output of the residual block - hence the name *residual*. Those blocks are much easier to train, and one can construct networks with several hundreds of those blocks (most common variants are ResNet-52, ResNet-101 and ResNet-152).
+
+You can also think of this network as being able to adjust its complexity to the dataset. Initially, when you are starting to train the network, the weights values are small, and most of the signal goes through passthrough identity layers. As training progresses and weights become larger, the significance of network parameters grow, and the networks adjusts to accommodate required expressive power to correctly classify training images.
+
+### Google Inception
+
+Google Inception architecture takes this idea one step further, and builds each network layer as a combination of several different paths:
+
+<img src="images/inception.png" width="400"/>
+
+> Image from [Researchgate](https://www.researchgate.net/figure/Inception-module-with-dimension-reductions-left-and-schema-for-Inception-ResNet-v1_fig2_355547454)
+
+Here, we need to emphasize the role of 1x1 convolutions, because at first they do not make sense. Why would we need to run through the image with 1x1 filter? However, you need to remember that convolution filters also work with several depth channels (originally - RGB colors, in subsequent layers - channels for different filters), and 1x1 convolution is used to mix those input channels together using different trainable weights. It can be also viewed as downsampling (pooling) over channel dimension.
+
+Here is [a good blog post](https://medium.com/analytics-vidhya/talented-mr-1x1-comprehensive-look-at-1x1-convolution-in-deep-learning-f6b355825578) on the subject, and [the original paper](https://arxiv.org/pdf/1312.4400.pdf).
+
+### MobileNet
+
+MobileNet is a family of models with reduced size, suitable for mobile devices. Use them if you are short in resources, and can sacrifice a little bit of accuracy. The main idea behind them is so-called **depthwise separable convolution**, which allows representing convolution filters by a composition of spatial convolutions and 1x1 convolution over depth channels. This significantly reduces the number of parameters, making the network smaller in size, and also easier to train with less data.
+
+Here is [a good blog post on MobileNet](https://medium.com/analytics-vidhya/image-classification-with-mobilenet-cc6fbb2cd470).
+
+## Conclusion
+
+In this unit, you have learned the main concept behind computer vision neural networks - convolutional networks. Real-life architectures that power image classification, object detection, and even image generation networks are all based on CNNs, just with more layers and some additional training tricks.
+
+## 🚀 Challenge
+
+In the accompanying notebooks, there are notes at the bottom about how to obtain greater accuracy. Do some experiments to see if you can achieve higher accuracy.
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/14)
+
+## Review & Self Study
+
+While CNNs are most often used for Computer Vision tasks, they are generally good for extracting fixed-sized patterns. For example, if we are dealing with sounds, we may also want to use CNNs to look for some specific patterns in audio signal - in which case filters would be 1-dimensional (and this CNN would be called 1D-CNN). Also, sometimes 3D-CNN is used to extract features in multi-dimensional space, such as certain events occurring on video - CNN can capture certain patterns of feature changing over time. Do some review and self-study about other tasks that can be done with CNNs.
+
+## [Assignment](lab/README.md)
+
+In this lab, you are tasked with classifying different cat and dog breeds. These images are more complex than the MNIST dataset and of higher dimensions, and there are more than 10 classes.

lessons/4-ComputerVision/07-ConvNets/README.md

@@ -0,0 +1,58 @@
+# Convolutional Neural Networks
+
+We have seen before that neural networks are quite good at dealing with images, and even one-layer perceptron is able to recognize handwritten digits from MNIST dataset with reasonable accuracy. However, the MNIST dataset is very special, and all digits are centered inside the image, which makes the task simpler.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/13)
+
+In real life, we want to be able to recognize objects on a picture regardless of their exact location in the image. Computer vision is different from generic classification, because when we are trying to find a certain object in the picture, we are scanning the image looking for some specific **patterns** and their combinations. For example, when looking for a cat, we first may look for horizontal lines, which can form whiskers, and then certain a combination of whiskers can tell us that it is actually a picture of a cat. Relative position and presence of certain patterns is important, and not their exact position on the image.
+
+To extract patterns, we will use the notion of **convolutional filters**. As you know, an image is represented by a 2D-matrix, or a 3D-tensor with color depth. Applying a filter means that we take relatively small **filter kernel** matrix, and for each pixel in the original image we compute the weighted average with neighboring points. We can view this like a small window sliding over the whole image, and averaging out all pixels according to the weights in the filter kernel matrix.
+
+![Vertical Edge Filter](images/filter-vert.png) | ![Horizontal Edge Filter](images/filter-horiz.png)
+----|----
+
+> Image by Dmitry Soshnikov
+
+For example, if we apply 3x3 vertical edge and horizontal edge filters to the MNIST digits, we can get highlights (e.g. high values) where there are vertical and horizontal edges in our original image. Thus those two filters can be used to "look for" edges. Similarly, we can design different filters to look for other low-level patterns:
+
+<img src="images/lmfilters.jpg" width="500" align="center"/>
+
+
+> Image of [Leung-Malik Filter Bank](https://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html)
+
+However, while we can design the filters to extract some patterns manually, we can also design the network in such a way that it will learn the patterns automatically. It is one of the main ideas behind the CNN.
+
+## Main ideas behind CNN
+
+The way CNNs work is based on the following important ideas:
+
+* Convolutional filters can extract patterns
+* We can design the network in such a way that filters are trained automatically
+* We can use the same approach to find patterns in high-level features, not only in the original image. Thus CNN feature extraction work on a hierarchy of features, starting from low-level pixel combinations, up to higher level combination of picture parts.
+
+![Hierarchical Feature Extraction](images/FeatureExtractionCNN.png)
+
+> Image from [a paper by Hislop-Lynch](https://www.semanticscholar.org/paper/Computer-vision-based-pedestrian-trajectory-Hislop-Lynch/26e6f74853fc9bbb7487b06dc2cf095d36c9021d), based on [their research](https://dl.acm.org/doi/abs/10.1145/1553374.1553453)
+
+## ✍️ Exercises: Convolutional Neural Networks
+
+Let's continue exploring how convolutional neural networks work, and how we can achieve trainable filters, by working through the corresponding notebooks:
+
+* [Convolutional Neural Networks - PyTorch](ConvNetsPyTorch.ipynb)
+* [Convolutional Neural Networks - TensorFlow](ConvNetsTF.ipynb)
+
+## Pyramid Architecture
+
+Most of the CNNs used for image processing follow a so-called pyramid architecture. The first convolutional layer applied to the original images typically has a relatively low number of filters (8-16), which correspond to different pixel combinations, such as horizontal/vertical lines of strokes. At the next level, we reduce the spatial dimension of the network, and increase the number of filters, which corresponds to more possible combinations of simple features. With each layer, as we move towards the final classifier, spatial dimensions of the image decrease, and the number of filters grow.
+
+As an example, let's look at the architecture of VGG-16, a network that achieved 92.7% accuracy in ImageNet's top-5 classification in 2014:
+
+![ImageNet Layers](images/vgg-16-arch1.jpg)
+
+![ImageNet Pyramid](images/vgg-16-arch.jpg)
+
+> Image from [Researchgate](https://www.researchgate.net/figure/Vgg16-model-structure-To-get-the-VGG-NIN-model-we-replace-the-2-nd-4-th-6-th-7-th_fig2_335194493)
+
+## Best-Known CNN Architectures
+
+[Continue your study about the best-known CNN architectures](CNN_Architectures.md)

lessons/4-ComputerVision/07-ConvNets/lab/README.md

@@ -0,0 +1,33 @@
+# Classification of Pets Faces
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+Imagine you need to develop and application for pet nursery to catalog all pets. One of the great features of such an application would be automatically discovering the breed from a photograph. This can be successfully done using neural networks.
+
+You need to train a convolutional neural network to classify different breeds of cats and dogs using **Pet Faces** dataset.
+
+## The Dataset
+
+We will use the [Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/), which contains images of 37 different breeds of dogs and cats.
+
+![Dataset we will deal with](images/data.png)
+
+To download the dataset, use this code snippet:
+
+```python
+!wget https://thor.robots.ox.ac.uk/~vgg/data/pets/images.tar.gz
+!tar xfz images.tar.gz
+!rm images.tar.gz
+```
+
+**Note:** The Oxford-IIIT Pet Dataset images are organized by filename (e.g., `Abyssinian_1.jpg`, `Bengal_2.jpg`). The notebook includes code to organize these images into breed-specific subdirectories for easier classification.
+
+## Stating Notebook
+
+Start the lab by opening [PetFaces.ipynb](PetFaces.ipynb)
+
+## Takeaway
+
+You have solved a relatively complex problem of image classification from scratch! There were quite a lot of classes, and you were still able to get reasonable accuracy! It also makes sense to measure top-k accuracy, because it is easy to confuse some of the classes which are not clearly different even to human beings.

lessons/4-ComputerVision/07-ConvNets/pytorchcv.py

@@ -0,0 +1,169 @@
+  
+# Script file to hide implementation details for PyTorch computer vision module
+
+import builtins
+import torch
+import torch.nn as nn
+from torch.utils import data
+import torchvision
+from torchvision.transforms import ToTensor
+import matplotlib.pyplot as plt
+import numpy as np
+from PIL import Image
+import glob
+import os
+import zipfile 
+
+default_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+def load_mnist(batch_size=64):
+    builtins.data_train = torchvision.datasets.MNIST('./data',
+        download=True,train=True,transform=ToTensor())
+    builtins.data_test = torchvision.datasets.MNIST('./data',
+        download=True,train=False,transform=ToTensor())
+    builtins.train_loader = torch.utils.data.DataLoader(data_train,batch_size=batch_size)
+    builtins.test_loader = torch.utils.data.DataLoader(data_test,batch_size=batch_size)
+    
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    net.train()
+    total_loss,acc,count = 0,0,0
+    for features,labels in dataloader:
+        optimizer.zero_grad()
+        lbls = labels.to(default_device)
+        out = net(features.to(default_device))
+        loss = loss_fn(out,lbls) #cross_entropy(out,labels)
+        loss.backward()
+        optimizer.step()
+        total_loss+=loss
+        _,predicted = torch.max(out,1)
+        acc+=(predicted==lbls).sum()
+        count+=len(labels)
+    return total_loss.item()/count, acc.item()/count
+
+def validate(net, dataloader,loss_fn=nn.NLLLoss()):
+    net.eval()
+    count,acc,loss = 0,0,0
+    with torch.no_grad():
+        for features,labels in dataloader:
+            lbls = labels.to(default_device)
+            out = net(features.to(default_device))
+            loss += loss_fn(out,lbls) 
+            pred = torch.max(out,1)[1]
+            acc += (pred==lbls).sum()
+            count += len(labels)
+    return loss.item()/count, acc.item()/count
+
+def train(net,train_loader,test_loader,optimizer=None,lr=0.01,epochs=10,loss_fn=nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    res = { 'train_loss' : [], 'train_acc': [], 'val_loss': [], 'val_acc': []}
+    for ep in range(epochs):
+        tl,ta = train_epoch(net,train_loader,optimizer=optimizer,lr=lr,loss_fn=loss_fn)
+        vl,va = validate(net,test_loader,loss_fn=loss_fn)
+        print(f"Epoch {ep:2}, Train acc={ta:.3f}, Val acc={va:.3f}, Train loss={tl:.3f}, Val loss={vl:.3f}")
+        res['train_loss'].append(tl)
+        res['train_acc'].append(ta)
+        res['val_loss'].append(vl)
+        res['val_acc'].append(va)
+    return res
+
+def train_long(net,train_loader,test_loader,epochs=5,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss(),print_freq=10):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    for epoch in range(epochs):
+        net.train()
+        total_loss,acc,count = 0,0,0
+        for i, (features,labels) in enumerate(train_loader):
+            lbls = labels.to(default_device)
+            optimizer.zero_grad()
+            out = net(features.to(default_device))
+            loss = loss_fn(out,lbls)
+            loss.backward()
+            optimizer.step()
+            total_loss+=loss
+            _,predicted = torch.max(out,1)
+            acc+=(predicted==lbls).sum()
+            count+=len(labels)
+            if i%print_freq==0:
+                print("Epoch {}, minibatch {}: train acc = {}, train loss = {}".format(epoch,i,acc.item()/count,total_loss.item()/count))
+        vl,va = validate(net,test_loader,loss_fn)
+        print("Epoch {} done, validation acc = {}, validation loss = {}".format(epoch,va,vl))
+
+
+def plot_results(hist):
+    plt.figure(figsize=(15,5))
+    plt.subplot(121)
+    plt.plot(hist['train_acc'], label='Training acc')
+    plt.plot(hist['val_acc'], label='Validation acc')
+    plt.legend()
+    plt.subplot(122)
+    plt.plot(hist['train_loss'], label='Training loss')
+    plt.plot(hist['val_loss'], label='Validation loss')
+    plt.legend()
+
+def plot_convolution(t,title=''):
+    with torch.no_grad():
+        c = nn.Conv2d(kernel_size=(3,3),out_channels=1,in_channels=1)
+        c.weight.copy_(t)
+        fig, ax = plt.subplots(2,6,figsize=(8,3))
+        fig.suptitle(title,fontsize=16)
+        for i in range(5):
+            im = data_train[i][0]
+            ax[0][i].imshow(im[0])
+            ax[1][i].imshow(c(im.unsqueeze(0))[0][0])
+            ax[0][i].axis('off')
+            ax[1][i].axis('off')
+        ax[0,5].imshow(t)
+        ax[0,5].axis('off')
+        ax[1,5].axis('off')
+        #plt.tight_layout()
+        plt.show()
+        
+def display_dataset(dataset, n=10,classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    mn = min([dataset[i][0].min() for i in range(n)])
+    mx = max([dataset[i][0].max() for i in range(n)])
+    for i in range(n):
+        ax[i].imshow(np.transpose((dataset[i][0]-mn)/(mx-mn),(1,2,0)))
+        ax[i].axis('off')
+        if classes:
+            ax[i].set_title(classes[dataset[i][1]])
+
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return True
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image: {}".format(fn))
+            os.remove(fn)
+
+
+def common_transform():
+    std_normalize = torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                          std=[0.229, 0.224, 0.225])
+    trans = torchvision.transforms.Compose([
+            torchvision.transforms.Resize(256),
+            torchvision.transforms.CenterCrop(224),
+            torchvision.transforms.ToTensor(), 
+            std_normalize])
+    return trans
+
+def load_cats_dogs_dataset():
+    if not os.path.exists('data/PetImages'):
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+
+    dataset = torchvision.datasets.ImageFolder('data/PetImages',transform=common_transform())
+    trainset, testset = torch.utils.data.random_split(dataset,[20000,len(dataset)-20000])
+    trainloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    testloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    return dataset, trainloader, testloader

lessons/4-ComputerVision/07-ConvNets/tfcv.py

@@ -0,0 +1,86 @@
+# Tensorflow Computer Vision Helper
+
+import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+import matplotlib.pyplot as plt
+from PIL import Image
+import glob
+import os
+
+def plot_convolution(data,t,title=''):
+    fig, ax = plt.subplots(2,len(data)+1,figsize=(8,3))
+    fig.suptitle(title,fontsize=16)
+    tt = np.expand_dims(np.expand_dims(t,2),2)
+    for i,im in enumerate(data):
+        ax[0][i].imshow(im)
+        ximg = np.expand_dims(np.expand_dims(im,2),0)
+        cim = tf.nn.conv2d(ximg,tt,1,'SAME')
+        ax[1][i].imshow(cim[0][:,:,0])
+        ax[0][i].axis('off')
+        ax[1][i].axis('off')
+    ax[0,-1].imshow(t)
+    ax[0,-1].axis('off')
+    ax[1,-1].axis('off')
+    #plt.tight_layout()
+    plt.show()
+
+def plot_results(hist):
+    fig,ax = plt.subplots(1,2,figsize=(15,3))
+    ax[0].set_title('Accuracy')
+    ax[1].set_title('Loss')
+    for x in ['acc','val_acc']:
+        ax[0].plot(hist.history[x])
+    for x in ['loss','val_loss']:
+        ax[1].plot(hist.history[x])
+    plt.show()
+
+def display_dataset(dataset, labels=None, n=10, classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    for i in range(n):
+        ax[i].imshow(dataset[i])
+        ax[i].axis('off')
+        if classes is not None and labels is not None:
+            ax[i].set_title(classes[labels[i][0]])
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return im.format=='JPEG'
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image or wrong format: {}".format(fn))
+            os.remove(fn)
+
+def load_cats_dogs_dataset(batch_size=64):
+    if not os.path.exists('data/PetImages'):
+        print("Extracting the dataset")
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+    print("Checking dataset")
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+    data_dir = 'data/PetImages'
+    print("Loading dataset")
+    ds_train = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'training',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    ds_test = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'validation',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    return ds_train,ds_test

lessons/4-ComputerVision/08-TransferLearning/README.md

@@ -0,0 +1,77 @@
+# Pre-trained Networks and Transfer Learning
+
+Training CNNs can take a lot of time, and a lot of data is required for that task. However, much of the time is spent learning the best low-level filters that a network can use to extract patterns from images. A natural question arises - can we use a neural network trained on one dataset and adapt it to classify different images without requiring a full training process?
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/15)
+
+This approach is called **transfer learning**, because we transfer some knowledge from one neural network model to another. In transfer learning, we typically start with a pre-trained model, which has been trained on some large image dataset, such as **ImageNet**. Those models can already do a good job extracting different features from generic images, and in many cases just building a classifier on top of those extracted features can yield a good result.
+
+> ✅ Transfer Learning is a term you find in other academic fields, such as Education. It refers to the process of taking knowledge from one domain and applying it to another.
+
+## Pre-Trained Models as Feature Extractors
+
+The convolutional networks that we have talked about in the previous section contained a number of layers, each of which is supposed to extract some features from the image, starting from low-level pixel combinations (such as horizontal/vertical line or stroke), up to higher level combinations of features, corresponding to things like an eye of a flame. If we train CNN on sufficiently large dataset of generic and diverse images, the network should learn to extract those common features.
+
+Both Keras and PyTorch contain functions to easily load pre-trained neural network weights for some common architectures, most of which were trained on ImageNet images. The most often used ones are described on the [CNN Architectures](../07-ConvNets/CNN_Architectures.md) page from the prior lesson. In particular, you may want to consider using one of the following:
+
+* **VGG-16/VGG-19** which are relatively simple models that still give good accuracy. Often using VGG as a first attempt is a good choice to see how transfer learning is working.
+* **ResNet** is a family of models proposed by Microsoft Research in 2015. They have more layers, and thus take more resources.
+* **MobileNet** is a family of models with reduced size, suitable for mobile devices. Use them if you are short in resources and can sacrifice a little bit of accuracy.
+
+Here are sample features extracted from a picture of a cat by VGG-16 network:
+
+![Features extracted by VGG-16](images/features.png)
+
+## Cats vs. Dogs Dataset
+
+In this example, we will use a dataset of [Cats and Dogs](https://www.microsoft.com/download/details.aspx?id=54765&WT.mc_id=academic-77998-cacaste), which is very close to a real-life image classification scenario.
+
+## ✍️ Exercise: Transfer Learning
+
+Let's see transfer learning in action in corresponding notebooks:
+
+* [Transfer Learning - PyTorch](TransferLearningPyTorch.ipynb)
+* [Transfer Learning - TensorFlow](TransferLearningTF.ipynb)
+
+## Visualizing Adversarial Cat
+
+Pre-trained neural network contains different patterns inside it's *brain*, including notions of **ideal cat** (as well as ideal dog, ideal zebra, etc.). It would be interesting to somehow **visualize this image**. However, it is not simple, because patterns are spread all over the network weights, and also organized in a hierarchical structure.
+
+One approach we can take is to start with a random image, and then try to use **gradient descent optimization** technique to adjust that image in such a way, that the network starts thinking that it's a cat. 
+
+![Image Optimization Loop](images/ideal-cat-loop.png)
+
+However, if we do this, we will receive something very similar to a random noise. This is because *there are many ways to make network think the input image is a cat*, including some that do not make sense visually. While those images contain a lot of patterns typical for a cat, there is nothing to constrain them to be visually distinctive.
+
+To improve the result, we can add another term into the loss function, which is called **variation loss**. It is a metric that shows how similar neighboring pixels of the image are. Minimizing variation loss makes image smoother, and gets rid of noise - thus revealing more visually appealing patterns. Here is an example of such "ideal" images, that are classified as cat and as zebra with high probability:
+
+![Ideal Cat](images/ideal-cat.png) | ![Ideal Zebra](images/ideal-zebra.png)
+-----|-----
+ *Ideal Cat* | *Ideal Zebra*
+
+Similar approach can be used to perform so-called **adversarial attacks** on a neural network. Suppose we want to fool a neural network and make a dog look like a cat. If we take dog's image, which is recognized by a network as a dog, we can then tweak it a little but using gradient descent optimization, until the network starts classifying it as a cat:
+
+![Picture of a Dog](images/original-dog.png) | ![Picture of a dog classified as a cat](images/adversarial-dog.png)
+-----|-----
+*Original picture of a dog* | *Picture of a dog classified as a cat*
+
+See the code to reproduce the results above in the following notebook:
+
+* [Ideal and Adversarial Cat - TensorFlow](AdversarialCat_TF.ipynb)
+## Conclusion
+
+Using transfer learning, you are able to quickly put together a classifier for a custom object classification task and achieve high accuracy. You can see that more complex tasks that we are solving now require higher computational power, and cannot be easily solved on the CPU. In the next unit, we will try to use a more lightweight implementation to train the same model using lower compute resources, which results in just slightly lower accuracy.
+
+## 🚀 Challenge
+
+In the accompanying notebooks, there are notes at the bottom about how transfer knowledge works best with somewhat similar training data (a new type of animal, perhaps). Do some experimentation with completely new types of images to see how well or poorly your transfer knowledge models perform.
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/16)
+
+## Review & Self Study
+
+Read through [TrainingTricks.md](TrainingTricks.md) to deepen your knowledge of some other way to train your models.
+
+## [Assignment](lab/README.md)
+
+In this lab, we will use real-life [Oxford-IIIT](https://www.robots.ox.ac.uk/~vgg/data/pets/) pets dataset with 35 breeds of cats and dogs, and we will build a transfer learning classifier.

lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md

@@ -0,0 +1,105 @@
+# Deep Learning Training Tricks
+
+As neural networks become deeper, the process of their training becomes more and more challenging. One major problem is so-called [vanishing gradients](https://en.wikipedia.org/wiki/Vanishing_gradient_problem) or [exploding gradients](https://deepai.org/machine-learning-glossary-and-terms/exploding-gradient-problem#:~:text=Exploding%20gradients%20are%20a%20problem,updates%20are%20small%20and%20controlled.). [This post](https://towardsdatascience.com/the-vanishing-exploding-gradient-problem-in-deep-neural-networks-191358470c11) gives a good introduction into those problems.
+
+To make training deep networks more efficient, there are a few techniques that can be used.
+
+## Keeping values in reasonable interval
+
+To make numerical computations more stable, we want to make sure that all values within our neural network are within reasonable scale, typically [-1..1] or [0..1]. It is not a very strict requirement, but the nature of floating point computations is such that values of different magnitudes cannot be accurately manipulated together. For example, if we add 10<sup>-10</sup> and 10<sup>10</sup>, we are likely to get 10<sup>10</sup>, because smaller value would be "converted" to the same order as the larger one, and thus mantissa would be lost.
+
+Most activation functions have non-linearities around [-1..1], and thus it makes sense to scale all input data to [-1..1] or [0..1] interval.
+
+## Initial Weight Initialization
+
+Ideally, we want the values to be in the same range after passing through network layers. Thus it is important to initialize weights in such a way as to preserve the distribution of values.
+
+Normal distribution **N(0,1)** is not a good idea, because if we have *n* inputs, the standard deviation of output would be *n*, and values are likely to jump out of [0..1] interval.
+
+The following initializations are often used:
+
+ * Uniform distribution -- `uniform`
+ * **N(0,1/n)** -- `gaussian`
+ * **N(0,1/&radic;n_in)** guarantees that for inputs with zero mean and standard deviation of 1 the same mean/standard deviation would remain
+ * **N(0,&radic;2/(n_in+n_out))** -- so-called **Xavier initialization** (`glorot`), it helps to keep the signals in range during both forward and backward propagation
+
+## Batch Normalization
+
+Even with proper weight initialization, weights can get arbitrary big or small during the training, and they will bring signals out of proper range. We can bring signals back by using one of **normalization** techniques. While there are several of them (Weight normalization, Layer Normalization), the most often used is Batch Normalization.
+
+The idea of **batch normalization** is to take into account all values across the minibatch, and perform normalization (i.e. subtract mean and divide by standard deviation) based on those values. It is implemented as a network layer that does this normalization after applying the weights, but before activation function. As a result, we are likely to see higher final accuracy and faster training.
+
+Here is the [original paper](https://arxiv.org/pdf/1502.03167.pdf) on batch normalization, the [explanation on Wikipedia](https://en.wikipedia.org/wiki/Batch_normalization), and [a good introductory blog post](https://towardsdatascience.com/batch-normalization-in-3-levels-of-understanding-14c2da90a338) (and the one [in Russian](https://habrahabr.ru/post/309302/)).
+
+## Dropout
+
+**Dropout** is an interesting technique that removes a certain percentage of random neurons during training. It is also implemented as a layer with one parameter (percentage of neurons to remove, typically 10%-50%), and during training it zeroes random elements of the input vector, before passing it to the next layer.
+
+While this may sound like a strange idea, you can see the effect of dropout on training MNIST digit classifier in [`Dropout.ipynb`](Dropout.ipynb) notebook. It speeds up training and allows us to achieve higher accuracy in less training epochs.
+
+This effect can be explained in several ways:
+
+ * It can be considered to be a random shocking factor to the model, which takes optimiation out of local minimum
+ * It can be considered as *implicit model averaging*, because we can say that during dropout we are training slightly different model
+
+> *Some people say that when a drunk person tries to learn something, he will remember this better next morning, comparing to a sober person, because a brain with some malfunctioning neurons tries to adapt better to gasp the meaning. We never tested ourselves if this is true of not*
+
+## Preventing overfitting
+
+One of the very important aspect of deep learning is too be able to prevent [overfitting](../../3-NeuralNetworks/05-Frameworks/Overfitting.md). While it might be tempting to use very powerful neural network model, we should always balance the number of model parameters with the number of training samples.
+
+> Make sure you understand the concept of [overfitting](../../3-NeuralNetworks/05-Frameworks/Overfitting.md) we have introduced earlier!
+
+There are several ways to prevent overfitting:
+
+ * Early stopping -- continuously monitor error on validation set and stopping training when validation error starts to increase.
+ * Explicit Weight Decay / Regularization -- adding an extra penalty to the loss function for high absolute values of weights, which prevents the model of getting very unstable results
+ * Model Averaging -- training several models and then averaging the result. This helps to minimize the variance.
+ * Dropout (Implicit Model Averaging)
+
+## Optimizers / Training Algorithms
+
+Another important aspect of training is to chose good training algorithm. While classical **gradient descent** is a reasonable choice, it can sometimes be too slow, or result in other problems.
+
+In deep learning, we use **Stochastic Gradient Descent** (SGD), which is a gradient descent applied to minibatches, randomly selected from the training set. Weights are adjusted using this formula:
+
+w<sup>t+1</sup> = w<sup>t</sup> - &eta;&nabla;&lagran;
+
+### Momentum
+
+In **momentum SGD**, we are keeping a portion of a gradient from previous steps. It is similar to when we are moving somewhere with inertia, and we receive a punch in a different direction, our trajectory does not change immediately, but keeps some part of the original movement. Here we introduce another vector v to represent the *speed*:
+
+* v<sup>t+1</sup> = &gamma; v<sup>t</sup> - &eta;&nabla;&lagran;
+* w<sup>t+1</sup> = w<sup>t</sup>+v<sup>t+1</sup>
+
+Here parameter &gamma; indicates the extent to which we take inertia into account: &gamma;=0 corresponds to classical SGD; &gamma;=1 is a pure motion equation.
+
+### Adam, Adagrad, etc.
+
+Since in each layer we multiply signals by some matrix W<sub>i</sub>, depending on ||W<sub>i</sub>||, the gradient can either diminish and be close to 0, or rise indefinitely. It is the essence of Exploding/Vanishing Gradients problem.
+
+One of the solutions to this problem is to use only direction of the gradient in the equation, and ignore the absolute value, i.e.
+
+w<sup>t+1</sup> = w<sup>t</sup> - &eta;(&nabla;&lagran;/||&nabla;&lagran;||), where ||&nabla;&lagran;|| = &radic;&sum;(&nabla;&lagran;)<sup>2</sup>
+
+This algorithm is called **Adagrad**. Another algorithms that use the same idea: **RMSProp**, **Adam**
+
+> **Adam** is considered to be a very efficient algorithm for many applications, so if you are not sure which one to use - use Adam.
+
+### Gradient clipping
+
+Gradient clipping is an extension the idea above. When the ||&nabla;&lagran;|| &le; &theta;, we consider the original gradient in the weight optimization, and when ||&nabla;&lagran;|| > &theta; - we divide the gradient by it's norm. Here &theta; is a parameter, in most cases we can take &theta;=1 or &theta;=10.
+
+### Learning rate decay
+
+Training success often depends on the learning rate parameter &eta;. It is logical to assume that larger values of &eta; result in faster training, which is something we typically want in the beginning of the training, and then smaller value of &eta; allow us to fine-tune the network. Thus, in most of the cases we want to decrease &eta; in the process of the training.
+
+This can be done by multiplying &eta; by some number (eg. 0.98) after each epoch of the training, or by using more complicated **learning rate schedule**.
+
+## Different Network Architectures
+
+Selecting right network architecture for your problem can be tricky. Normally, we would take an architecture that has proven to work for our specific task (or similar one). Here is a [good overview](https://www.topbots.com/a-brief-history-of-neural-network-architectures/) or neural network architectures for computer vision.
+
+> It is important to select an architecture that will be powerful enough for the number of training samples that we have. Selecting too powerful model can result in [overfitting](../../3-NeuralNetworks/05-Frameworks/Overfitting.md)
+
+Another good way would be to use and architecture that will automatically adjust to the required complexity. To some extent, **ResNet** architecture and **Inception** are self-adjusting. [More on computer vision architectures](../07-ConvNets/CNN_Architectures.md)

lessons/4-ComputerVision/08-TransferLearning/lab/README.md

@@ -0,0 +1,27 @@
+# Classification of Oxford Pets using Transfer Learning
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+Imagine you need to develop and application for pet nursery to catalog all pets. One of the great features of such an application would be automatically discovering the breed from a photograph. In this assignment, we will use transfer learning to classify real-life pet images from [Oxford-IIIT](https://www.robots.ox.ac.uk/~vgg/data/pets/) pets dataset.
+
+## The Dataset
+
+We will use the original [Oxford-IIIT](https://www.robots.ox.ac.uk/~vgg/data/pets/) pets dataset, which contains 35 different breeds of dogs and cats.
+
+To download the dataset, use this code snippet:
+
+```python
+!wget https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz
+!tar xfz images.tar.gz
+!rm images.tar.gz
+```
+
+## Stating Notebook
+
+Start the lab by opening [OxfordPets.ipynb](OxfordPets.ipynb)
+
+## Takeaway
+
+Transfer learning and pre-trained networks allow us to solve real-world image classification problems relatively easily. However, pre-trained networks work well on images of similar kind, and if we start classifying very different images (eg. medical images), we are likely to get much worse results.

lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py

@@ -0,0 +1,169 @@
+  
+# Script file to hide implementation details for PyTorch computer vision module
+
+import builtins
+import torch
+import torch.nn as nn
+from torch.utils import data
+import torchvision
+from torchvision.transforms import ToTensor
+import matplotlib.pyplot as plt
+import numpy as np
+from PIL import Image
+import glob
+import os
+import zipfile 
+
+default_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+def load_mnist(batch_size=64):
+    builtins.data_train = torchvision.datasets.MNIST('./data',
+        download=True,train=True,transform=ToTensor())
+    builtins.data_test = torchvision.datasets.MNIST('./data',
+        download=True,train=False,transform=ToTensor())
+    builtins.train_loader = torch.utils.data.DataLoader(data_train,batch_size=batch_size)
+    builtins.test_loader = torch.utils.data.DataLoader(data_test,batch_size=batch_size)
+    
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    net.train()
+    total_loss,acc,count = 0,0,0
+    for features,labels in dataloader:
+        optimizer.zero_grad()
+        lbls = labels.to(default_device)
+        out = net(features.to(default_device))
+        loss = loss_fn(out,lbls) #cross_entropy(out,labels)
+        loss.backward()
+        optimizer.step()
+        total_loss+=loss
+        _,predicted = torch.max(out,1)
+        acc+=(predicted==lbls).sum()
+        count+=len(labels)
+    return total_loss.item()/count, acc.item()/count
+
+def validate(net, dataloader,loss_fn=nn.NLLLoss()):
+    net.eval()
+    count,acc,loss = 0,0,0
+    with torch.no_grad():
+        for features,labels in dataloader:
+            lbls = labels.to(default_device)
+            out = net(features.to(default_device))
+            loss += loss_fn(out,lbls) 
+            pred = torch.max(out,1)[1]
+            acc += (pred==lbls).sum()
+            count += len(labels)
+    return loss.item()/count, acc.item()/count
+
+def train(net,train_loader,test_loader,optimizer=None,lr=0.01,epochs=10,loss_fn=nn.NLLLoss()):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    res = { 'train_loss' : [], 'train_acc': [], 'val_loss': [], 'val_acc': []}
+    for ep in range(epochs):
+        tl,ta = train_epoch(net,train_loader,optimizer=optimizer,lr=lr,loss_fn=loss_fn)
+        vl,va = validate(net,test_loader,loss_fn=loss_fn)
+        print(f"Epoch {ep:2}, Train acc={ta:.3f}, Val acc={va:.3f}, Train loss={tl:.3f}, Val loss={vl:.3f}")
+        res['train_loss'].append(tl)
+        res['train_acc'].append(ta)
+        res['val_loss'].append(vl)
+        res['val_acc'].append(va)
+    return res
+
+def train_long(net,train_loader,test_loader,epochs=5,lr=0.01,optimizer=None,loss_fn = nn.NLLLoss(),print_freq=10):
+    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+    for epoch in range(epochs):
+        net.train()
+        total_loss,acc,count = 0,0,0
+        for i, (features,labels) in enumerate(train_loader):
+            lbls = labels.to(default_device)
+            optimizer.zero_grad()
+            out = net(features.to(default_device))
+            loss = loss_fn(out,lbls)
+            loss.backward()
+            optimizer.step()
+            total_loss+=loss
+            _,predicted = torch.max(out,1)
+            acc+=(predicted==lbls).sum()
+            count+=len(labels)
+            if i%print_freq==0:
+                print("Epoch {}, minibatch {}: train acc = {}, train loss = {}".format(epoch,i,acc.item()/count,total_loss.item()/count))
+        vl,va = validate(net,test_loader,loss_fn)
+        print("Epoch {} done, validation acc = {}, validation loss = {}".format(epoch,va,vl))
+
+
+def plot_results(hist):
+    plt.figure(figsize=(15,5))
+    plt.subplot(121)
+    plt.plot(hist['train_acc'], label='Training acc')
+    plt.plot(hist['val_acc'], label='Validation acc')
+    plt.legend()
+    plt.subplot(122)
+    plt.plot(hist['train_loss'], label='Training loss')
+    plt.plot(hist['val_loss'], label='Validation loss')
+    plt.legend()
+
+def plot_convolution(t,title=''):
+    with torch.no_grad():
+        c = nn.Conv2d(kernel_size=(3,3),out_channels=1,in_channels=1)
+        c.weight.copy_(t)
+        fig, ax = plt.subplots(2,6,figsize=(8,3))
+        fig.suptitle(title,fontsize=16)
+        for i in range(5):
+            im = data_train[i][0]
+            ax[0][i].imshow(im[0])
+            ax[1][i].imshow(c(im.unsqueeze(0))[0][0])
+            ax[0][i].axis('off')
+            ax[1][i].axis('off')
+        ax[0,5].imshow(t)
+        ax[0,5].axis('off')
+        ax[1,5].axis('off')
+        #plt.tight_layout()
+        plt.show()
+        
+def display_dataset(dataset, n=10,classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    mn = min([dataset[i][0].min() for i in range(n)])
+    mx = max([dataset[i][0].max() for i in range(n)])
+    for i in range(n):
+        ax[i].imshow(np.transpose((dataset[i][0]-mn)/(mx-mn),(1,2,0)))
+        ax[i].axis('off')
+        if classes:
+            ax[i].set_title(classes[dataset[i][1]])
+
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return True
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image: {}".format(fn))
+            os.remove(fn)
+
+
+def common_transform():
+    std_normalize = torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                          std=[0.229, 0.224, 0.225])
+    trans = torchvision.transforms.Compose([
+            torchvision.transforms.Resize(256),
+            torchvision.transforms.CenterCrop(224),
+            torchvision.transforms.ToTensor(), 
+            std_normalize])
+    return trans
+
+def load_cats_dogs_dataset():
+    if not os.path.exists('data/PetImages'):
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+
+    dataset = torchvision.datasets.ImageFolder('data/PetImages',transform=common_transform())
+    trainset, testset = torch.utils.data.random_split(dataset,[20000,len(dataset)-20000])
+    trainloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    testloader = torch.utils.data.DataLoader(trainset,batch_size=32)
+    return dataset, trainloader, testloader

lessons/4-ComputerVision/08-TransferLearning/tfcv.py

@@ -0,0 +1,86 @@
+# Tensorflow Computer Vision Helper
+
+import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+import matplotlib.pyplot as plt
+from PIL import Image
+import glob
+import os
+
+def plot_convolution(data,t,title=''):
+    fig, ax = plt.subplots(2,len(data)+1,figsize=(8,3))
+    fig.suptitle(title,fontsize=16)
+    tt = np.expand_dims(np.expand_dims(t,2),2)
+    for i,im in enumerate(data):
+        ax[0][i].imshow(im)
+        ximg = np.expand_dims(np.expand_dims(im,2),0)
+        cim = tf.nn.conv2d(ximg,tt,1,'SAME')
+        ax[1][i].imshow(cim[0][:,:,0])
+        ax[0][i].axis('off')
+        ax[1][i].axis('off')
+    ax[0,-1].imshow(t)
+    ax[0,-1].axis('off')
+    ax[1,-1].axis('off')
+    #plt.tight_layout()
+    plt.show()
+
+def plot_results(hist):
+    fig,ax = plt.subplots(1,2,figsize=(15,3))
+    ax[0].set_title('Accuracy')
+    ax[1].set_title('Loss')
+    for x in ['acc','val_acc']:
+        ax[0].plot(hist.history[x])
+    for x in ['loss','val_loss']:
+        ax[1].plot(hist.history[x])
+    plt.show()
+
+def display_dataset(dataset, labels=None, n=10, classes=None):
+    fig,ax = plt.subplots(1,n,figsize=(15,3))
+    for i in range(n):
+        ax[i].imshow(dataset[i])
+        ax[i].axis('off')
+        if classes is not None and labels is not None:
+            ax[i].set_title(classes[labels[i][0]])
+
+def check_image(fn):
+    try:
+        im = Image.open(fn)
+        im.verify()
+        return im.format=='JPEG'
+    except:
+        return False
+    
+def check_image_dir(path):
+    for fn in glob.glob(path):
+        if not check_image(fn):
+            print("Corrupt image or wrong format: {}".format(fn))
+            os.remove(fn)
+
+def load_cats_dogs_dataset(batch_size=64):
+    if not os.path.exists('data/PetImages'):
+        print("Extracting the dataset")
+        with zipfile.ZipFile('data/kagglecatsanddogs_3367a.zip', 'r') as zip_ref:
+            zip_ref.extractall('data')
+    print("Checking dataset")
+    check_image_dir('data/PetImages/Cat/*.jpg')
+    check_image_dir('data/PetImages/Dog/*.jpg')
+    data_dir = 'data/PetImages'
+    print("Loading dataset")
+    ds_train = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'training',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    ds_test = keras.preprocessing.image_dataset_from_directory(
+        data_dir,
+        validation_split = 0.2,
+        subset = 'validation',
+        seed = 13,
+        image_size = (224,224),
+        batch_size = batch_size
+    )
+    return ds_train,ds_test

lessons/4-ComputerVision/09-Autoencoders/README.md

@@ -0,0 +1,97 @@
+# Autoencoders
+
+When training CNNs, one of the problems is that we need a lot of labeled data. In the case of image classification, we need to separate images into different classes, which is a manual effort.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/17)
+
+However, we might want to use raw (unlabeled) data for training CNN feature extractors, which is called **self-supervised learning**. Instead of labels, we will use training images as both network input and output. The main idea of **autoencoder** is that we will have an **encoder network** that converts input image into some **latent space** (normally it is just a vector of some smaller size), then the **decoder network**, whose goal would be to reconstruct the original image.
+
+> ✅ An [autoencoder](https://wikipedia.org/wiki/Autoencoder) is "a type of artificial neural network used to learn efficient codings of unlabeled data."
+
+Since we are training an autoencoder to capture as much of the information from the original image as possible for accurate reconstruction, the network tries to find the best **embedding** of input images to capture the meaning.л.
+
+![AutoEncoder Diagram](images/autoencoder_schema.jpg)
+
+> Image from [Keras blog](https://blog.keras.io/building-autoencoders-in-keras.html)
+
+## Scenarios for using Autoencoders
+
+While reconstructing original images does not seem useful in its own right, there are a few scenarios where autoencoders are especially useful:
+
+* **Lowering the dimension of images for visualization** or **training image embeddings**. Usually autoencoders give better results than PCA, because it takes into account spatial nature of images and hierarchical features.
+* **Denoising**, i.e. removing noise from the image. Because noise carries out a lot of useless information, autoencoder cannot fit it all into relatively small latent space, and thus it captures only important part of the image. When training denoisers, we start with original images, and use images with artificially added noise as input for autoencoder.
+* **Super-resolution**, increasing image resolution. We start with high-resolution images, and use the image with lower resolution as the autoencoder input.
+* **Generative models**. Once we train the autoencoder, the decoder part can be used to create new objects starting from random latent vectors.
+
+## Variational Autoencoders (VAE)
+
+Traditional autoencoders reduce the dimension of the input data somehow, figuring out the important features of input images. However, latent vectors ofter do not make much sense. In other words, taking MNIST dataset as an example, figuring out which digits correspond to different latent vectors is not an easy task, because close latent vectors would not necessarily correspond to the same digits.
+
+On the other hand, to train *generative* models it is better to have some understanding of the latent space. This idea leads us to **variational auto-encoder** (VAE).
+
+VAE is the autoencoder that learns to predict *statistical distribution* of the latent parameters, so-called **latent distribution**. For example, we may want latent vectors to be distributed normally with some mean z<sub>mean</sub> and standard deviation z<sub>sigma</sub> (both mean and standard deviation are vectors of some dimensionality d). Encoder in VAE learns to predict those parameters, and then decoder takes a random vector from this distribution to reconstruct the object.
+
+To summarize:
+
+ * From input vector, we predict `z_mean` and `z_log_sigma` (instead of predicting the standard deviation itself, we predict its logarithm)
+ * We sample a vector `sample` from the distribution N(z<sub>mean</sub>,exp(z<sub>log\_sigma</sub>))
+ * The decoder tries to decode the original image using `sample` as an input vector
+
+ <img src="images/vae.png" width="50%">
+
+> Image from [this blog post](https://ijdykeman.github.io/ml/2016/12/21/cvae.html) by Isaak Dykeman
+
+Variational auto-encoders use a complex loss function that consists of two parts:
+
+* **Reconstruction loss** is the loss function that shows how close a reconstructed image is to the target (it can be Mean Squared Error, or MSE). It is the same loss function as in normal autoencoders.
+* **KL loss**, which ensures that latent variable distributions stays close to normal distribution. It is based on the notion of [Kullback-Leibler divergence](https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained) - a metric to estimate how similar two statistical distributions are.
+
+One important advantage of VAEs is that they allow us to generate new images relatively easily, because we know which distribution from which to sample latent vectors. For example, if we train VAE with 2D latent vector on MNIST, we can then vary components of the latent vector to get different digits:
+
+<img alt="vaemnist" src="images/vaemnist.png" width="50%"/>
+
+> Image by [Dmitry Soshnikov](http://soshnikov.com)
+
+Observe how images blend into each other, as we start getting latent vectors from the different portions of the latent parameter space. We can also visualize this space in 2D:
+
+<img alt="vaemnist cluster" src="images/vaemnist-diag.png" width="50%"/> 
+
+> Image by [Dmitry Soshnikov](http://soshnikov.com)
+
+## ✍️ Exercises: Autoencoders
+
+Learn more about autoencoders in these corresponding notebooks:
+
+* [Autoencoders in TensorFlow](AutoencodersTF.ipynb)
+* [Autoencoders in PyTorch](AutoEncodersPyTorch.ipynb)
+
+## Properties of Autoencoders
+
+* **Data Specific** - they only work well with the type of images they have been trained on. For example, if we train a super-resolution network on flowers, it will not work well on portraits. This is because the network can produce higher resolution image by taking fine details from features learned from the training dataset.
+* **Lossy** - the reconstructed image is not the same as the original image. The nature of loss is defined by the *loss function* used during training
+* Works on **unlabeled data**
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/18)
+
+## Conclusion
+
+In this lesson, you learned about the various types of autoencoders available to the AI scientist. You learned how to build them, and how to use them to reconstruct images. You also learned about the VAE and how to use it to generate new images.
+
+## 🚀 Challenge
+
+In this lesson, you learned about using autoencoders for images. But they can also be used for music! Check out the Magenta project's [MusicVAE](https://magenta.tensorflow.org/music-vae) project, which uses autoencoders to learn to reconstruct music. Do some [experiments](https://colab.research.google.com/github/magenta/magenta-demos/blob/master/colab-notebooks/Multitrack_MusicVAE.ipynb) with this library to see what you can create.
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/16)
+
+## Review & Self Study
+
+For reference, read more about autoencoders in these resources:
+
+* [Building Autoencoders in Keras](https://blog.keras.io/building-autoencoders-in-keras.html)
+* [Blog post on NeuroHive](https://neurohive.io/ru/osnovy-data-science/variacionnyj-avtojenkoder-vae/)
+* [Variational Autoencoders Explained](https://kvfrans.com/variational-autoencoders-explained/)
+* [Conditional Variational Autoencoders](https://ijdykeman.github.io/ml/2016/12/21/cvae.html)
+
+## Assignment
+
+At the end of [this notebook using TensorFlow](AutoencodersTF.ipynb), you will find a 'task' - use this as your assignment.

lessons/4-ComputerVision/10-GANs/README.md

@@ -0,0 +1,98 @@
+# Generative Adversarial Networks
+
+In the previous section, we learned about **generative models**: models that can generate new images similar to the ones in the training dataset. VAE was a good example of a generative model.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/19)
+
+However, if we try to generate something really meaningful, like a painting at reasonable resolution, with VAE, we will see that training does not converge well. For this use case, we should learn about another architecture specifically targeted at generative models - **Generative Adversarial Networks**, or GANs.
+
+The main idea of a GAN is to have two neural networks that will be trained against each other:
+
+<img src="images/gan_architecture.png" width="70%"/>
+
+> Image by [Dmitry Soshnikov](http://soshnikov.com)
+
+> ✅ A little vocabulary:
+> * **Generator** is a network that takes some random vector, and produces the image as a result
+> * **Discriminator** is a network that takes an image, and it should tell whether it is a real image (from training dataset), or it was generated by a generator. It is essentially an image classifier.
+
+### Discriminator
+
+The architecture of discriminator does not differ from an ordinary image classification network. In the simplest case it can be fully-connected classifier, but most probably it will be a [convolutional network](../07-ConvNets/README.md).
+
+> ✅ A GAN based on convolutional networks is called a [DCGAN](https://arxiv.org/pdf/1511.06434.pdf)
+
+A CNN discriminator consists of the following layers: several convolutions+poolings (with decreasing spatial size) and, one-or-more fully-connected layers to get "feature vector", final binary classifier.
+
+> ✅ A 'pooling' in this context is a technique that reduces the size of the image. "Pooling layers reduce the dimensions of data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer." - [source](https://wikipedia.org/wiki/Convolutional_neural_network#Pooling_layers)
+
+### Generator
+
+A Generator is slightly more tricky. You can consider it to be a reversed discriminator. Starting from a latent vector (in place of a feature vector), it has a fully-connected layer to convert it into the required size/shape, followed by deconvolutions+upscaling. This is similar to *decoder* part of [autoencoder](../09-Autoencoders/README.md).
+
+> ✅ Because the convolution layer is implemented as a linear filter traversing the image, deconvolution is essentially similar to convolution, and can be implemented using the same layer logic.
+
+<img src="images/gan_arch_detail.png" width="70%"/>
+
+> Image by [Dmitry Soshnikov](http://soshnikov.com)
+
+### Training the GAN
+
+GANs are called **adversarial** because there is a constant competition between the generator and the discriminator. During this competition, both generator and discriminator improve, thus the network learns to produce better and better pictures.
+
+The training happens in two stages:
+
+* **Training the discriminator**. This task is pretty straightforward: we generate a batch of images by the generator, labeling them 0, which stands for fake image, and taking a batch of images from the input dataset (with label 1, real image). We obtain some *discriminator loss*, and perform backprop.
+* **Training the generator**. This is slightly more tricky, because we do not know the expected output for the generator directly. We take the whole GAN network consisting of a generator followed by discriminator, feed it with some random vectors, and expect the result to be 1 (corresponding to real images). We then freeze the parameters of the discriminator (we do not want it to be trained at this step), and perform the backprop.
+
+During this process, both the generator and the discriminator losses are not going down significantly. In the ideal situation, they should oscillate, corresponding to both networks improving their performance.
+
+## ✍️ Exercises: GANs
+
+* [GAN Notebook in TensorFlow/Keras](GANTF.ipynb)
+* [GAN Notebook in PyTorch](GANPyTorch.ipynb)
+
+### Problems with GAN training
+
+GANs are known to be especially difficult to train. Here are a few problems:
+
+* **Mode Collapse**. By this term we mean that the generator learns to produce one successful image that tricks the generator, and not a variety of different images.
+* **Sensitivity to hyperparameters**. Often you can see that a GAN does not converge at all, and then suddenly decreases in the learning rate leading to convergence.
+* Keeping a **balance** between the generator and the discriminator. In many cases discriminator loss can drop to zero relatively quickly, which results in the generator being unable to train further. To overcome this, we can try setting different learning rates for the generator and discriminator, or skip discriminator training if the loss is already too low.
+* Training for **high resolution**. Reflecting the same problem as with autoencoders, this problem is triggered because reconstructing too many layers of convolutional network leads to artifacts. This problem is typically solved with so-called **progressive growing**, when first a few layers are trained on low-res images, and then layers are "unblocked" or added. Another solution would be adding extra connections between layers and training several resolutions at once - see this [Multi-Scale Gradient GANs paper](https://arxiv.org/abs/1903.06048) for details.
+
+## Style Transfer
+
+GANs is a great way to generate artistic images. Another interesting technique is so-called **style transfer**, which takes one **content image**, and re-draws it in a different style, applying filters from **style image**. 
+
+The way it works is the following:
+* We start with a random noise image (or with a content image, but for the sake of understanding it is easier to start from random noise)
+* Our goal would be to create such an image, that would be close to both content image and style image. This would be determined by two loss functions:
+   - **Content loss** is computed based on the features extracted by the CNN at some layers from current image and content image
+   - **Style loss** is computed between current image and style image in a clever way using Gram matrices (more details in the [example notebook](StyleTransfer.ipynb))
+* To make the image smoother and remove noise, we also introduce **Variation loss**, which computes average distance between neighboring pixels
+* The main optimization loop adjusts current image using gradient descent (or some other optimization algorithm) to minimize the total loss, which is a weighted sum of all three losses. 
+
+## ✍️ Example: [Style Transfer](StyleTransfer.ipynb)
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/20)
+
+## Conclusion
+
+In this lesson, you learned about GANS and how to train them. You also learned about the special challenges that this type of Neural Network can face, and some strategies on how to move past them.
+
+## 🚀 Challenge
+
+Run through the [Style Transfer notebook](StyleTransfer.ipynb) using your own images.
+
+## Review & Self Study
+
+For reference, read more about GANs in these resources:
+
+* Marco Pasini, [10 Lessons I Learned Training GANs for one Year](https://towardsdatascience.com/10-lessons-i-learned-training-generative-adversarial-networks-gans-for-a-year-c9071159628)
+* [StyleGAN](https://en.wikipedia.org/wiki/StyleGAN), a *de facto* GAN architecture to consider
+* [Creating Generative Art using GANs on Azure ML](https://soshnikov.com/scienceart/creating-generative-art-using-gan-on-azureml/)
+
+## Assignment
+
+Revisit one of the two notebooks associated to this lesson and retrain the GAN on your own images. What can you create?

lessons/4-ComputerVision/11-ObjectDetection/README.md

@@ -0,0 +1,176 @@
+# Object Detection
+
+The image classification models we have dealt with so far took an image and produced a categorical result, such as the class 'number' in a MNIST problem. However, in many cases we do not want just to know that a picture portrays objects - we want to be able to determine their precise location. This is exactly the point of **object detection**.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/21)
+
+![Object Detection](images/Screen_Shot_2016-11-17_at_11.14.54_AM.png)
+
+> Image from [YOLO v2 web site](https://pjreddie.com/darknet/yolov2/)
+
+## A Naive Approach to Object Detection
+
+Assuming we wanted to find a cat on a picture, a very naive approach to object detection would be the following:
+
+1. Break the picture down to a number of tiles
+2. Run image classification on each tile.
+3. Those tiles that result in sufficiently high activation can be considered to contain the object in question.
+
+![Naive Object Detection](images/naive-detection.png)
+
+> *Image from [Exercise Notebook](ObjectDetection-TF.ipynb)*
+
+However, this approach is far from ideal, because it only allows the algorithm to locate the object's bounding box very imprecisely. For more precise location, we need to run some sort of **regression** to predict the coordinates of bounding boxes - and for that, we need specific datasets.
+
+## Regression for Object Detection
+
+[This blog post](https://towardsdatascience.com/object-detection-with-neural-networks-a4e2c46b4491) has a great gentle introduction to detecting shapes.
+
+## Datasets for Object Detection
+
+You might run across the following datasets for this task:
+
+* [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/) - 20 classes
+* [COCO](http://cocodataset.org/#home) - Common Objects in Context. 80 classes, bounding boxes and segmentation masks
+
+![COCO](images/coco-examples.jpg)
+
+## Object Detection Metrics
+
+### Intersection over Union
+
+While for image classification it is easy to measure how well the algorithm performs, for object detection we need to measure both the correctness of the class, as well as the precision of the inferred bounding box location. For the latter, we use the so-called **Intersection over Union** (IoU), which measures how well two boxes (or two arbitrary areas) overlap.
+
+![IoU](images/iou_equation.png)
+
+> *Figure 2 from [this excellent blog post on IoU](https://pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/)*
+
+The idea is simple - we divide the area of intersection between two figures by the area of their union. For two identical areas, IoU would be 1, while for completely disjointed areas it will be 0. Otherwise it will vary from 0 to 1. We typically only consider those bounding boxes for which IoU is over a certain value.
+
+### Average Precision
+
+Suppose we want to measure how well a given class of objects $C$ is recognized. To measure it, we use **Average Precision** metrics, which is calculated as follows:
+
+1. Consider Precision-Recall curve shows the accuracy depending on a detection threshold value (from 0 to 1).
+2. Depending on the threshold, we will get more or less objects detected in the image, and different values of precision and recall.
+3. The curve will look like this:
+
+<img src="https://github.com/shwars/NeuroWorkshop/raw/master/images/ObjDetectionPrecisionRecall.png"/>
+
+> *Image from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop)*
+
+The average Precision for a given class $C$ is the area under this curve. More precisely, Recall axis is typically divided into 10 parts, and Precision is averaged over all those points:
+
+$$
+AP = {1\over11}\sum_{i=0}^{10}\mbox{Precision}(\mbox{Recall}={i\over10})
+$$
+
+### AP and IoU
+
+We shall consider only those detections, for which IoU is above a certain value. For example, in PASCAL VOC dataset typically $\mbox{IoU Threshold} = 0.5$ is assumed, while in COCO AP is measured for different values of $\mbox{IoU Threshold}$.
+
+<img src="https://github.com/shwars/NeuroWorkshop/raw/master/images/ObjDetectionPrecisionRecallIoU.png"/>
+
+> *Image from [NeuroWorkshop](http://github.com/shwars/NeuroWorkshop)*
+
+### Mean Average Precision - mAP
+
+The main metric for Object Detection is called **Mean Average Precision**, or **mAP**. It is the value of Average Precision, average across all object classes, and sometimes also over $\mbox{IoU Threshold}$. In more detail, the process of calculating **mAP** is described
+[in this blog post](https://medium.com/@timothycarlen/understanding-the-map-evaluation-metric-for-object-detection-a07fe6962cf3)), and also [here with code samples](https://gist.github.com/tarlen5/008809c3decf19313de216b9208f3734).
+
+## Different Object Detection Approaches
+
+There are two broad classes of object detection algorithms:
+
+* **Region Proposal Networks** (R-CNN, Fast R-CNN, Faster R-CNN). The main idea is to generate **Regions of Interests** (ROI) and run CNN over them, looking for maximum activation. It is a bit similar to the naive approach, with the exception that ROIs are generated in a more clever way. One of the majors drawbacks of such methods is that they are slow, because we need many passes of the CNN classifier over the image.
+* **One-pass** (YOLO, SSD, RetinaNet) methods. In those architectures we design the network to predict both classes and ROIs in one pass.
+
+### R-CNN: Region-Based CNN
+
+[R-CNN](http://islab.ulsan.ac.kr/files/announcement/513/rcnn_pami.pdf) uses [Selective Search](http://www.huppelen.nl/publications/selectiveSearchDraft.pdf) to generate hierarchical structure of ROI regions, which are then passed through CNN feature extractors and SVM-classifiers to determine the object class, and linear regression to determine *bounding box* coordinates. [Official Paper](https://arxiv.org/pdf/1506.01497v1.pdf)
+
+![RCNN](images/rcnn1.png)
+
+> *Image from van de Sande et al. ICCV’11*
+
+![RCNN-1](images/rcnn2.png)
+
+> *Images from [this blog](https://towardsdatascience.com/r-cnn-fast-r-cnn-faster-r-cnn-yolo-object-detection-algorithms-36d53571365e)
+
+### F-RCNN - Fast R-CNN
+
+This approach is similar to R-CNN, but regions are defined after convolution layers have been applied.
+
+![FRCNN](images/f-rcnn.png)
+
+> Image from [the Official Paper](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Girshick_Fast_R-CNN_ICCV_2015_paper.pdf), [arXiv](https://arxiv.org/pdf/1504.08083.pdf), 2015
+
+### Faster R-CNN
+
+The main idea of this approach is to use neural network to predict ROIs - so-called *Region Proposal Network*. [Paper](https://arxiv.org/pdf/1506.01497.pdf), 2016
+
+![FasterRCNN](images/faster-rcnn.png)
+
+> Image from [the official paper](https://arxiv.org/pdf/1506.01497.pdf)
+
+### R-FCN: Region-Based Fully Convolutional Network
+
+This algorithm is even faster than Faster R-CNN. The main idea is the following:
+
+1. We extract features using ResNet-101
+1. Features are processed by **Position-Sensitive Score Map**. Each object from $C$ classes is divided by $k\times k$ regions, and we are training to predict parts of objects.
+1. For each part from $k\times k$ regions all networks vote for object classes, and the object class with maximum vote is selected.
+
+![r-fcn image](images/r-fcn.png)
+
+> Image from [official paper](https://arxiv.org/abs/1605.06409)
+
+### YOLO - You Only Look Once
+
+YOLO is a realtime one-pass algorithm. The main idea is the following:
+
+ * Image is divided into $S\times S$ regions
+ * For each region, **CNN** predicts $n$ possible objects, *bounding box* coordinates and *confidence*=*probability* * IoU.
+
+ ![YOLO](images/yolo.png)
+
+> Image from [official paper](https://arxiv.org/abs/1506.02640)
+
+### Other Algorithms
+
+* RetinaNet: [official paper](https://arxiv.org/abs/1708.02002)
+   - [PyTorch Implementation in Torchvision](https://pytorch.org/vision/stable/_modules/torchvision/models/detection/retinanet.html)
+   - [Keras Implementation](https://github.com/fizyr/keras-retinanet)
+   - [Object Detection with RetinaNet](https://keras.io/examples/vision/retinanet/) in Keras Samples
+* SSD (Single Shot Detector): [official paper](https://arxiv.org/abs/1512.02325)
+
+## ✍️ Exercises: Object Detection
+
+Continue your learning in the following notebook:
+
+[ObjectDetection.ipynb](ObjectDetection.ipynb)
+
+## Conclusion
+
+In this lesson you took a whirlwind tour of all the various ways that object detection can be accomplished!
+
+## 🚀 Challenge
+
+Read through these articles and notebooks about YOLO and try them for yourself
+
+* [Good blog post](https://www.analyticsvidhya.com/blog/2018/12/practical-guide-object-detection-yolo-framewor-python/) describing YOLO
+ * [Official site](https://pjreddie.com/darknet/yolo/)
+ * Yolo: [Keras implementation](https://github.com/experiencor/keras-yolo2), [step-by-step notebook](https://github.com/experiencor/basic-yolo-keras/blob/master/Yolo%20Step-by-Step.ipynb)
+ * Yolo v2: [Keras implementation](https://github.com/experiencor/keras-yolo2), [step-by-step notebook](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/22)
+
+## Review & Self Study
+
+* [Object Detection](https://tjmachinelearning.com/lectures/1718/obj/) by Nikhil Sardana
+* [A good comparison of object detection algorithms](https://lilianweng.github.io/lil-log/2018/12/27/object-detection-part-4.html)
+* [Review of Deep Learning Algorithms for Object Detection](https://medium.com/comet-app/review-of-deep-learning-algorithms-for-object-detection-c1f3d437b852)
+* [A Step-by-Step Introduction to the Basic Object Detection Algorithms](https://www.analyticsvidhya.com/blog/2018/10/a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1/)
+* [Implementation of Faster R-CNN in Python for Object Detection](https://www.analyticsvidhya.com/blog/2018/11/implementation-faster-r-cnn-python-object-detection/)
+
+## [Assignment: Object Detection](lab/README.md)

lessons/4-ComputerVision/11-ObjectDetection/lab/README.md

@@ -0,0 +1,63 @@
+# Head Detection using Hollywood Heads Dataset
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+Counting number of people on video surveillance camera stream is an important task that will allow us to estimate the number of visitors in a shops, busy hours in a restaurant, etc. To solve this task, we need to be able to detect human heads from different angles. To train object detection model to detect human heads, we can use [Hollywood Heads Dataset](https://www.di.ens.fr/willow/research/headdetection/).
+
+## The Dataset
+
+[Hollywood Heads Dataset](https://www.di.ens.fr/willow/research/headdetection/release/HollywoodHeads.zip) contains 369,846 human heads annotated in 224,740 movie frames from Hollywood movies. It is provided in [https://host.robots.ox.ac.uk/pascal/VOC/](PASCAL VOC) format, where for each image there is also an XML description file that looks like this:
+
+```xml
+<annotation>
+	<folder>HollywoodHeads</folder>
+	<filename>mov_021_149390.jpeg</filename>
+	<source>
+		<database>HollywoodHeads 2015 Database</database>
+		<annotation>HollywoodHeads 2015</annotation>
+		<image>WILLOW</image>
+	</source>
+	<size>
+		<width>608</width>
+		<height>320</height>
+		<depth>3</depth>
+	</size>
+	<segmented>0</segmented>
+	<object>
+		<name>head</name>
+		<bndbox>
+			<xmin>201</xmin>
+			<ymin>1</ymin>
+			<xmax>480</xmax>
+			<ymax>263</ymax>
+		</bndbox>
+		<difficult>0</difficult>
+	</object>
+	<object>
+		<name>head</name>
+		<bndbox>
+			<xmin>3</xmin>
+			<ymin>4</ymin>
+			<xmax>241</xmax>
+			<ymax>285</ymax>
+		</bndbox>
+		<difficult>0</difficult>
+	</object>
+</annotation>
+```
+
+In this dataset, there is only one class of objects `head`, and for each head, you get the coordinates of the bounding box. You can parse XML using Python libraries, or use [this library](https://pypi.org/project/pascal-voc/) to deal directly with PASCAL VOC format.
+
+## Training Object Detection 
+
+You can train an object detection model using one of the following ways:
+
+* Using [Azure Custom Vision](https://docs.microsoft.com/azure/cognitive-services/custom-vision-service/quickstarts/object-detection?tabs=visual-studio&WT.mc_id=academic-77998-cacaste) and it's Python API to programmatically train the model in the cloud. Custom vision will not be able to use more than a few hundred images for training the model, so you may need to limit the dataset.
+* Using the example from [Keras tutorial](https://keras.io/examples/vision/retinanet/) to train RetunaNet model.
+* Using [torchvision.models.detection.RetinaNet](https://pytorch.org/vision/stable/_modules/torchvision/models/detection/retinanet.html) build-in module in torchvision.
+
+## Takeaway
+
+Object detection is a task that is frequently required in industry. While there are some services that can be used to perform object detection (such as [Azure Custom Vision](https://docs.microsoft.com/azure/cognitive-services/custom-vision-service/quickstarts/object-detection?tabs=visual-studio&WT.mc_id=academic-77998-cacaste)), it is important to understand how object detection works and to be able to train your own models. 

lessons/4-ComputerVision/12-Segmentation/README.md

@@ -0,0 +1,66 @@
+# Segmentation
+
+We have previously learned about Object Detection, which allows us to locate objects in the image by predicting their *bounding boxes*. However, for some tasks we do not only need bounding boxes, but also more precise object localization. This task is called  **segmentation**.
+
+## [Pre-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/23)
+
+Segmentation can be viewed as **pixel classification**, whereas for **each** pixel of image we must predict its class (*background* being one of the classes). There are two main segmentation algorithms:
+
+* **Semantic segmentation** only tells the pixel class, and does not make a distinction between different objects of the same class
+* **Instance segmentation** divides classes into different instances.
+
+For instance segmentation, these sheep are different objects, but for semantic segmentation all sheep are represented by one class.
+
+<img src="images/instance_vs_semantic.jpeg" width="50%">
+
+> Image from [this blog post](https://nirmalamurali.medium.com/image-classification-vs-semantic-segmentation-vs-instance-segmentation-625c33a08d50)
+
+There are different neural architectures for segmentation, but they all have the same structure. In a way, it is similar to the autoencoder you learned about previously, but instead of deconstructing the original image, our goal is to deconstruct a **mask**. Thus, a segmentation network has the following parts:
+
+* **Encoder** extracts features from input image
+* **Decoder** transforms those features into the **mask image**, with the same size and number of channels corresponding to the number of classes.
+
+<img src="images/segm.png" width="80%">
+
+> Image from [this publication](https://arxiv.org/pdf/2001.05566.pdf)
+
+We should especially mention the loss function that is used for segmentation. When using classical autoencoders, we need to measure the similarity between two images, and we can use mean square error (MSE) to do that. In segmentation, each pixel in the target mask image represents the class number (one-hot-encoded along the third dimension), so we need to use loss functions specific for classification - cross-entropy loss, averaged over all pixels. If the mask is binary - **binary cross-entropy loss** (BCE) is used.
+
+> ✅ One-hot encoding is a way to encode a class label into a vector of length equal to the number of classes. Take a look at [this article](https://datagy.io/sklearn-one-hot-encode/) on this technique.
+
+## Segmentation for Medical Imaging
+
+In this lesson, we will see the segmentation in action by training the network to recognize human nevi (also known as moles) on medical images. We will be using <a href="https://www.fc.up.pt/addi/ph2%20database.html">PH<sup>2</sup> Database</a> of dermoscopy images as the image source. This dataset contains 200 images of three classes: typical nevus, atypical nevus, and melanoma. All images also contain a corresponding **mask** that outlines the nevus.
+
+> ✅ This technique is particularly appropriate for this type of medical imaging, but what other real-world applications could you envision?
+
+<img alt="navi" src="images/navi.png"/>
+
+> Image from the PH<sup>2</sup> Database
+
+We will train a model to segment any nevus from its background.
+
+## ✍️ Exercises: Semantic Segmentation
+
+Open the notebooks below to learn more about different semantic segmentation architectures, practice working with them, and see them in action.
+
+* [Semantic Segmentation Pytorch](SemanticSegmentationPytorch.ipynb)
+* [Semantic Segmentation TensorFlow](SemanticSegmentationTF.ipynb)
+
+## [Post-lecture quiz](https://ff-quizzes.netlify.app/en/ai/quiz/24)
+
+## Conclusion
+
+Segmentation is a very powerful technique for image classification, moving beyond bounding boxes to pixel-level classification. It is a technique used in medical imaging, among other applications.
+
+## 🚀 Challenge
+
+Body segmentation is just one of the common tasks that we can do with images of people. Another important tasks include **skeleton detection** and **pose detection**. Try out [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) library to see how pose detection can be used.
+
+## Review & Self Study
+
+This [wikipedia article](https://wikipedia.org/wiki/Image_segmentation) offers a good overview of the various applications of this technique. Learn more on your own about the subdomains of Instance segmentation and Panoptic segmentation in this field of inquiry.
+
+## [Assignment](lab/README.md)
+
+In this lab, try **human body segmentation** using [Segmentation Full Body MADS Dataset](https://www.kaggle.com/datasets/tapakah68/segmentation-full-body-mads-dataset) from Kaggle.

lessons/4-ComputerVision/12-Segmentation/lab/README.md

@@ -0,0 +1,19 @@
+# Human Body Segmentation
+
+Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners).
+
+## Task
+
+In video production, for example, in weather forecasts, we often need to cut out a human image from camera and place it on top of some other footage. This is typically done using **chroma key** techniques, when a human is filmed in front of a uniform color background, which is then removed. In this lab, we will train a neural network model to cut out the human silhouette.
+
+## The Dataset
+
+We will be using [Segmentation Full Body MADS Dataset](https://www.kaggle.com/datasets/tapakah68/segmentation-full-body-mads-dataset) from Kaggle. Download the dataset manually from Kaggle.
+
+## Stating Notebook
+
+Start the lab by opening [BodySegmentation.ipynb](BodySegmentation.ipynb)
+
+## Takeaway
+
+Body segmentation is just one of the common tasks that we can do with images of people. Another important tasks include **skeleton detection** and **pose detection**. Look into [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) library to see how those tasks can be implemented.

lessons/4-ComputerVision/README.md

@@ -0,0 +1,13 @@
+# Computer Vision
+
+![Summary of Computer Vision content in a doodle](../sketchnotes/ai-computervision.png)
+
+In this section we will learn about:
+
+* [Intro to Computer Vision and OpenCV](06-IntroCV/README.md)
+* [Convolutional Neural Networks](07-ConvNets/README.md)
+* [Pre-trained Networks and Transfer Learning](08-TransferLearning/README.md) 
+* [Autoencoders](09-Autoencoders/README.md)
+* [Generative Adversarial Networks](10-GANs/README.md)
+* [Object Detection](11-ObjectDetection/README.md)
+* [Semantic Segmentation](12-Segmentation/README.md)