diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..a762cf8 --- /dev/null +++ b/README.en.md @@ -0,0 +1,427 @@ + +# Deep Learning Models + +A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks. + + + +## Traditional Machine Learning + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Perceptron | 2D toy data | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/basic-ml/perceptron.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/basic-ml/perceptron.ipynb) | +| Logistic Regression | 2D toy data | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/basic-ml/logistic-regression.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/basic-ml/logistic-regression.ipynb)| +| Softmax Regression (Multinomial Logistic Regression) | MNIST | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/basic-ml/softmax-regression.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/basic-ml/softmax-regression.ipynb) | +| Softmax Regression with MLxtend's plot_decision_regions on Iris | Iris | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/basic-ml/softmax-regression-mlxtend-1.ipynb) | + + +## Multilayer Perceptrons + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Multilayer Perceptron | MNIST | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/mlp/mlp-basic.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mlp/mlp-basic.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mlp/mlp-basic.ipynb) | +| Multilayer Perceptron with Dropout | MNIST | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/mlp/mlp-dropout.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mlp/mlp-dropout.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mlp/mlp-dropout.ipynb) | +|Multilayer Perceptron with Batch Normalization | MNIST | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/mlp/mlp-batchnorm.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mlp/mlp-batchnorm.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mlp/mlp-batchtnorm.ipynb) | +|Multilayer Perceptron with Backpropagation from Scratch | MNIST | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mlp/mlp-fromscratch__sigmoid-mse.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mlp/mlp-fromscratch__sigmoid-mse.ipynb)| + + + +## Convolutional Neural Networks + + +#### Basic + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Convolutional Neural Network | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-basic.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-basic.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/cnn/cnn-basic.ipynb) | +| CNN with He Initialization | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-he-init.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-he-init.ipynb) | + + + +#### Concepts + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Replacing Fully-Connected by Equivalent Convolutional Layers | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/fc-to-conv.ipynb) | + + + +--- + + +#### AlexNet + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| AlexNet Trained on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-alexnet-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-alexnet-cifar10.ipynb) | +| AlexNet with Grouped Convolutions Trained on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-alexnet-grouped-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-alexnet-grouped-cifar10.ipynb) | + + + +#### DenseNet + +|Title | Description | Daset | Notebooks | +| --- | --- | --- | --- | +| DenseNet-121 Digit Classifier Trained on MNIST | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-densenet121-mnist.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-densenet121-mnist.ipynb) | +| DenseNet-121 Image Classifier Trained on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-densenet121-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-densenet121-cifar10.ipynb) | + + + +#### Fully Convolutional + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| "All Convolutionl Net" -- A Fully Convolutional Neural Network | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-allconv.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-allconv.ipynb) | + +#### LeNet + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| LeNet-5 on MNIST | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-lenet5-mnist.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-lenet5-mnist.ipynb) | +| LeNet-5 on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-lenet5-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-lenet5-cifar10.ipynb) | +| LeNet-5 on QuickDraw | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-lenet5-quickdraw.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-lenet5-quickdraw.ipynb) | + + + + +#### MobileNet + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| MobileNet-v2 on Cifar-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v2-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-mobilenet-v2-cifar10.ipynb) | +| MobileNet-v3 small on Cifar-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v3-small-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-mobilenet-v3-small-cifar10.ipynb) | +| MobileNet-v3 large on Cifar-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-mobilenet-v3-large-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-mobilenet-v3-large-cifar10.ipynb) | +| MobileNet-v3 large on MNIST via Embetter | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-embetter-mobilenet.ipynb) | + + + + +#### Network in Network + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Network in Network Trained on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-nin-cifar10.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/nin-cifar10.ipynb) | + + +#### VGG + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Convolutional Neural Network VGG-16 Trained on CIFAR-10 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-vgg16.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-vgg16.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/cnn/cnn-vgg16.ipynb) | +| VGG-16 Smile Classifier | [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-vgg16-celeba.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-vgg16-celeba.ipynb) | +| VGG-16 Dogs vs Cats Classifier | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-vgg16-cats-dogs.ipynb) | +| Convolutional Neural Network VGG-19 | TBD | TBD | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/cnn/cnn-vgg19.ipynb) [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-vgg19.ipynb) | + + + + +#### ResNet + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| ResNet and Residual Blocks | [MNIST](http://yann.lecun.com/exdb/mnist/) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/resnet-ex-1.ipynb) | +| ResNet-18 Digit Classifier| [MNIST](http://yann.lecun.com/exdb/mnist/) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet18-mnist.ipynb) | +| ResNet-18 Gender Classifier | [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet18-celeba-dataparallel.ipynb) | +| ResNet-34 Digit Classifier | [MNIST](http://yann.lecun.com/exdb/mnist/) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet34-mnist.ipynb) | +| ResNet-34 Object Classifier | [QuickDraw](https://quickdraw.withgoogle.com) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet34-quickdraw.ipynb) | +| ResNet-34 Gender Classifier| [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet34-celeba-dataparallel.ipynb) | +| ResNet-50 Digit Classifier| [MNIST](http://yann.lecun.com/exdb/mnist/) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet50-mnist.ipynb) | +| ResNet-50 Gender Classifier | [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet50-celeba-dataparallel.ipynb) | +| ResNet-101 Gender Classifier| [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet101-celeba.ipynb) | +| ResNet-101| [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet101-cifar10.ipynb) | +| ResNet-152 Gender Classifier| [CelebA](https://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet152-celeba.ipynb) | + + +--- + +## Transformers + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Multilabel DistilBERT | [Jigsaw Toxic Comment Challenge](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge) | DistilBERT classifier fine-tuning | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/transformer/distilbert-multilabel.ipynb) | +| DistilBERT as feature extractor | [IMDB movie review](https://ai.stanford.edu/~amaas/data/sentiment/) | DistilBERT classifier with sklearn random forest and logistic regression | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/transformer/1_distilbert-as-feature-extractor.ipynb) | +| DistilBERT as feature extractor using `embetter` | [IMDB movie review](https://ai.stanford.edu/~amaas/data/sentiment/) | DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn `embetter` library | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/transformer/distilbert-embetter-feature-extractor.ipynb) | +| Fine-tune DistilBERT I | [IMDB movie review](https://ai.stanford.edu/~amaas/data/sentiment/) | Fine-tune only the last 2 layers of DistilBERT classifier | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/transformer/distilbert-finetune-last-layers.ipynb) | +| Fine-tune DistilBERT II | [IMDB movie review](https://ai.stanford.edu/~amaas/data/sentiment/) | Fine-tune the whole DistilBERT classifier | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/transformer/distilbert-hf-finetuning.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/transformer/distilbert-finetuning-ii.ipynb) | + +--- + +## Ordinal Regression and Deep Learning + +Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks. + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Baseline multilayer perceptron | Cement | A baseline multilayer perceptron for classification trained with the standard cross entropy loss | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/baseline_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/baseline-light_cement.ipynb) | +| CORAL multilayer perceptron | Cement | Implementation of [Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation](https://www.sciencedirect.com/science/article/pii/S016786552030413X) 2020 | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/CORAL_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/CORAL-light_cement.ipynb) | +| CORN multilayer perceptron | Cement | Implementation of [Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities](https://arxiv.org/abs/2111.08851) 2022 | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/CORN_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/CORN-light_cement.ipynb) | +| Binary extension multilayer perceptron | Cement | Implementation of [Ordinal Regression with Multiple Output CNN for Age Estimation](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Niu_Ordinal_Regression_With_CVPR_2016_paper.pdf) 2016 | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/niu2016_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/niu2016-light_cement.ipynb) | +| Reformulated squared-error multilayer perceptron | Cement | Implementation of [A simple squared-error reformulation for ordinal classification](https://arxiv.org/abs/1612.00775) 2016 | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/beckham2016_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/beckham2016-light_cement.ipynb) | +| Class distance weighted cross-entropy loss | Cement | Implementation of [Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation](https://arxiv.org/abs/2202.05167) 2022 | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/ordinal/polat2022_cement.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/ordinal/polat2022-light_cement.ipynb) | + +--- + + + +## Normalization Layers + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/nin-cifar10_batchnorm.ipynb) | +| Filter Response Normalization for Network-in-Network CIFAR-10 Classifier | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/nin-cifar10_filter-response-norm.ipynb) | + + + +## Metric Learning + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Siamese Network with Multilayer Perceptrons | TBD | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/metric/siamese-1.ipynb) | + +## Autoencoders + +#### Fully-connected Autoencoders + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Autoencoder (MNIST) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-basic.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/autoencoder/ae-basic.ipynb) | +| Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-basic-with-rf.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/autoencoder/ae-basic-with-rf.ipynb) | + + + + + + +#### Convolutional Autoencoders + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Convolutional Autoencoder with Deconvolutions / Transposed Convolutions | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-deconv.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/autoencoder/ae-deconv.ipynb) | +| Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-deconv-jaccard.ipynb) | +| Convolutional Autoencoder with Deconvolutions (without pooling operations) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-deconv-nopool.ipynb) | +| Convolutional Autoencoder with Nearest-neighbor Interpolation | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-conv-nneighbor.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/autoencoder/ae-conv-nneighbor.ipynb) | +| Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-conv-nneighbor-celeba.ipynb) | +| Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-conv-nneighbor-quickdraw-1.ipynb) | + + + +#### Variational Autoencoders + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Variational Autoencoder | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-var.ipynb) | +| Convolutional Variational Autoencoder | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-conv-var.ipynb) | + + + +#### Conditional Variational Autoencoders + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Conditional Variational Autoencoder (with labels in reconstruction loss) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-cvae.ipynb) | +| Conditional Variational Autoencoder (without labels in reconstruction loss) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-cvae_no-out-concat.ipynb) | +| Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-cnn-cvae.ipynb) | +| Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/autoencoder/ae-cnn-cvae_no-out-concat.ipynb) | + + + + +## Generative Adversarial Networks (GANs) + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Fully Connected GAN on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/gan.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/gan/gan.ipynb) | +| Fully Connected Wasserstein GAN on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/wgan-1.ipynb) | +| Convolutional GAN on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/gan-conv.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/gan/gan-conv.ipynb) | +| Convolutional GAN on MNIST with Label Smoothing | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/gan-conv-smoothing.ipynb) [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/gan/gan-conv-smoothing.ipynb) | +| Convolutional Wasserstein GAN on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/dc-wgan-1.ipynb) | +| Deep Convolutional GAN (DCGAN) on Cats and Dogs Images | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/dcgan-cats-and-dogs.ipynb) | +| Deep Convolutional GAN (DCGAN) on CelebA Face Images | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gan/dcgan-celeba.ipynb) | + + +## Graph Neural Networks (GNNs) + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Most Basic Graph Neural Network with Gaussian Filter on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gnn/gnn-basic-1.ipynb) | +| Basic Graph Neural Network with Edge Prediction on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gnn/gnn-basic-edge-1.ipynb) | +| Basic Graph Neural Network with Spectral Graph Convolution on MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/gnn/gnn-basic-graph-spectral-1.ipynb) | + + + +## Recurrent Neural Networks (RNNs) + + + + +#### Many-to-one: Sentiment Analysis / Classification + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| A simple single-layer RNN (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_simple_imdb.ipynb) | +| A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_simple_packed_imdb.ipynb) | +| RNN with LSTM cells (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_lstm_packed_imdb.ipynb) | +| RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_lstm_packed_imdb-glove.ipynb) | +| RNN with LSTM cells and Own Dataset in CSV Format (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_lstm_packed_own_csv_imdb.ipynb) | +| RNN with GRU cells (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_gru_packed_imdb.ipynb) | +| Multilayer bi-directional RNN (IMDB) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_lstm_bi_imdb.ipynb) | +| Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_bi_multilayer_lstm_own_csv_agnews.ipynb) | + + + +#### Many-to-Many / Sequence-to-Sequence + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| A simple character RNN to generate new text (Charles Dickens) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/char_rnn-charlesdickens.ipynb) | + + + +## Model Evaluation + +### K-Fold Cross-Validation + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Baseline CNN | MNIST | A simple baseline with traditional train/validation/test splits | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/kfold/baseline-cnn-mnist.ipynb) [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/kfold/baseline-light-cnn-mnist.ipynb) | +| K-fold with `pl_cross` | MNIST | A 5-fold cross-validation run using the `pl_cross` library | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](pytorch-lightning_ipynb/kfold/kfold-light-cnn-mnist.ipynb) | + + + +## Data Augmentation + +| Title | Dataset | Description | Notebooks | +| -------------------------- | ------- | ----------- | ------------------------------------------------------------ | +| AutoAugment & TrivialAugment for Image Data | CIFAR-10 | Trains a ResNet-18 using AutoAugment and TrivialAugment | [![PyTorch Lightning](https://img.shields.io/badge/PyTorch-Lightning-blueviolet)](./pytorch-lightning_ipynb/data-augmentation/autoaugment) | + + + + +## Tips and Tricks + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Cyclical Learning Rate | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/tricks/cyclical-learning-rate.ipynb) | +| Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/tricks/cnn-alexnet-cifar10-batchincrease.ipynb) | +| Gradient Clipping (w. MLP on MNIST) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/tricks/gradclipping_mlp.ipynb) | + + + + +## Transfer Learning + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/transfer/transferlearning-vgg16-cifar10-1.ipynb) | + + +## Visualization and Interpretation + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/viz/cnns/cats-and-dogs/cnn-viz-grad__vgg16-cats-dogs.ipynb) | +| Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/viz/cnns/cats-and-dogs/cnn-viz-guided-backprop__vgg16-cats-dogs.ipynb) | + + + +## PyTorch Workflows and Mechanics + + +#### PyTorch Lightning Examples + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| MLP in Lightning with TensorBoard -- continue training the last model | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/lightning/lightning-mlp.ipynb) | +| MLP in Lightning with TensorBoard -- checkpointing best model | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/lightning/lightning-mlp-best-model) | + + + + +#### Custom Datasets + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Custom Data Loader Example for PNG Files | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-dataloader-png/custom-dataloader-example.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-csv.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-celeba.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-quickdraw.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-svhn.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-afad.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader_dating-historical-color-images.ipynb) | +| Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/custom-data-loader-quickdraw.ipynb) | + + + +#### Training and Preprocessing + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| PyTorch DataLoader State and Nested Iterations | Toy | Explains DataLoader behavior when in nested functions | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/dataloader-nesting.ipynb)| +| Generating Validation Set Splits | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/validation-splits.ipynb) | +| Dataloading with Pinned Memory | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-resnet34-cifar10-pinmem.ipynb) | +| Standardizing Images | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-standardized.ipynb) | +| Image Transformation Examples | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/torchvision-transform-examples.ipynb) | +| Char-RNN with Own Text File | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/char_rnn-charlesdickens.ipynb) | +| Sentiment Classification RNN with Own CSV File | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/rnn/rnn_lstm_packed_own_csv_imdb.ipynb) | + + + +#### Improving Memory Efficiency + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/gradient-checkpointing-nin.ipynb) | + +#### Parallel Computing + +|Title | Description | Notebooks | +| --- | --- | --- | +| Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/cnn/cnn-vgg16-celeba-data-parallel.ipynb) | +| Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/model-pipeline-vgg16.ipynb) | + + +#### Other + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| PyTorch with and without Deterministic Behavior -- Runtime Benchmark | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/deterministic_benchmark.ipynb) | +| Sequential API and hooks | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/mlp-sequential.ipynb) | +| Weight Sharing Within a Layer | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/cnn-weight-sharing.ipynb) | +| Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/plot-jupyter-matplotlib.ipynb) | + + + +#### Autograd + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Getting Gradients of an Intermediate Variable in PyTorch | TBD | TBD | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/mechanics/manual-gradients.ipynb) | + + +## TensorFlow Workflows and Mechanics + +#### Custom Datasets + +|Title | Description | Notebooks | +| --- | --- | --- | +| Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/image-data-chunking-npz.ipynb) | +| Storing an Image Dataset for Minibatch Training using HDF5 | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/image-data-chunking-hdf5.ipynb) | +| Using Input Pipelines to Read Data from TFRecords Files | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/tfrecords.ipynb) | +| Using Queue Runners to Feed Images Directly from Disk | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/file-queues.ipynb) | +| Using TensorFlow's Dataset API | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/dataset-api.ipynb) | + + + + +#### Training and Preprocessing + +|Title | Dataset | Description | Notebooks | +| --- | --- | --- | --- | +| Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives | TBD | TBD | [![TensorFlow](https://img.shields.io/badge/Tensor-Flow1.0-orange)](tensorflow1_ipynb/mechanics/saving-and-reloading-models.ipynb) | + +## Related Libraries + +|Title | Description | Notebooks | +| --- | --- | --- | +| TorchMetrics | How do we use it, and what's the difference between .update() and .forward()? | [![PyTorch](https://img.shields.io/badge/Py-Torch-red)](pytorch_ipynb/related-libraries/torchmetrics-update-forward.ipynb) | +