chore: import upstream snapshot with attribution

2026-07-13 13:38:23 +08:00
commit 2725b63d23
700 changed files with 53581 additions and 0 deletions
@@ -0,0 +1,174 @@
+# Python Machine Learning FAQ
+
+It is always a pleasure to engage in discussions all around machine learning and I am happy to answer any questions regarding this book.
+
+I just thought that it might be worthwhile to compile some documents about
+the most commonly asked questions to answer them more thoroughly.
+
+Just drop me your question, feedback, or suggestion via your medium of choice and it will be answered :)
+
+- [mailing List](https://groups.google.com/forum/#!forum/python-machine-learning-book)
+- [private email](mailto:mail@sebastianraschka.com)
+- [twitter](https://twitter.com/rasbt)
+- [GitHub Issue Tracker](https://github.com/rasbt/python-machine-learning-book/issues)
+
+Cheers,
+Sebastian
+
+## FAQ
+
+<!--- start -->
+
+### General Questions about Machine Learning and 'Data Science'
+
+- [What are machine learning and data science?](./datascience-ml.md)
+- [Why do you and other people sometimes implement machine learning algorithms from scratch?](./implementing-from-scratch.md)
+- [What learning path/discipline in data science I should focus on?](./data-science-career.md)
+- [At what point should one start contributing to open source?](./open-source.md)
+- [How important do you think having a mentor is to the learning process?](./mentor.md)
+- [Where are the best online communities centered around data science/machine learning or python?](./ml-python-communities.md)
+- [How would you explain machine learning to a software engineer?](./ml-to-a-programmer.md)
+- [How would your curriculum for a machine learning beginner look like?](./ml-curriculum.md)
+- [What is the Definition of Data Science?](./definition_data-science.md)
+- [How do Data Scientists perform model selection? Is it different from Kaggle?](./model-selection-in-datascience.md)
+
+### Questions about the Machine Learning Field
+
+- [How are Artificial Intelligence and Machine Learning related?](./ai-and-ml.md)
+- [What are some real-world examples of applications of machine learning in the field?](./ml-examples.md)
+- [What are the different fields of study in data mining?](./datamining-overview.md)
+- [What are differences in research nature between the two fields: machine learning & data mining?](./datamining-vs-ml.md)
+- [How do I know if the problem is solvable through machine learning?](./ml-solvable.md)
+- [What are the origins of machine learning?](./ml-origins.md)
+- [How was classification, as a learning machine, developed?](./classifier-history.md)
+- [Which machine learning algorithms can be considered as among the best?](./best-ml-algo.md)
+- [What are the broad categories of classifiers?](./classifier-categories.md)
+- [What is the difference between a classifier and a model?](./difference_classifier_model.md)
+- [What is the difference between a parametric learning algorithm and a nonparametric learning algorithm?](./parametric_vs_nonparametric.md)
+- [What is the difference between a cost function and a loss function in machine learning?](./cost-vs-loss.md)
+
+### Questions about Machine Learning Concepts and Statistics
+
+##### Cost Functions and Optimization
+
+- [Fitting a model via closed-form equations vs. Gradient Descent vs Stochastic Gradient Descent vs Mini-Batch Learning -- what is the difference?](./closed-form-vs-gd.md)
+- [How do you derive the Gradient Descent rule for Linear Regression and Adaline?](./linear-gradient-derivative.md)
+
+##### Regression Analysis
+
+- [What is the difference between Pearson R and Simple Linear Regression?](./pearson-r-vs-linear-regr.md)
+
+##### Tree models
+
+- [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](./bagging-boosting-rf.md)
+- [What are the disadvantages of using classic decision tree algorithm for a large dataset?](./decision-tree-disadvantages.md)
+- [Why are implementations of decision tree algorithms usually binary, and what are the advantages of the different impurity metrics?](./decision-tree-binary.md)
+- [Why are we growing decision trees via entropy instead of the classification error?](./decisiontree-error-vs-entropy.md)
+- [When can a random forest perform terribly?](./random-forest-perform-terribly.md)
+
+##### Model evaluation
+
+- [What is overfitting?](./overfitting.md)
+- [How can I avoid overfitting?](./avoid-overfitting.md)
+- [Is it always better to have the largest possible number of folds when performing cross validation?](./number-of-kfolds.md)
+- [When training an SVM classifier, is it better to have a large or small number of support vectors?](./num-support-vectors.md)
+- [How do I evaluate a model?](./evaluate-a-model.md)
+- [What is the best validation metric for multi-class classification?](./multiclass-metric.md)
+- [What factors should I consider when choosing a predictive model technique?](./choosing-technique.md)
+- [What are the best toy datasets to help visualize and understand classifier behavior?](./clf-behavior-data.md)
+- [How do I select SVM kernels?](./select_svm_kernels.md)
+- [Interlude: Comparing and Computing Performance Metrics in Cross-Validation -- Imbalanced Class Problems and 3 Different Ways to Compute the F1 Score](./computing-the-f1-score.md)
+
+
+##### Logistic Regression
+
+- [What is Softmax regression and how is it related to Logistic regression?](./softmax_regression.md)
+- [Why is logistic regression considered a linear model?](./logistic_regression_linear.md)
+- [What is the probabilistic interpretation of regularized logistic regression?](./probablistic-logistic-regression.md)
+- [Does regularization in logistic regression always results in better fit and better generalization?](./regularized-logistic-regression-performance.md)
+- [What is the major difference between naive Bayes and logistic regression?](./naive-bayes-vs-logistic-regression.md)
+- [What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning?](./softmax.md)
+- [What is the relation between Loigistic Regression and Neural Networks and when to use which?](./logisticregr-neuralnet.md)
+- [Logistic Regression: Why sigmoid function?](./logistic-why-sigmoid.md)
+- [Is there an analytical solution to Logistic Regression similar to the Normal Equation for Linear Regression?](./logistic-analytical.md)
+
+##### Neural Networks and Deep Learning
+
+- [What is the difference between deep learning and usual machine learning?](./difference-deep-and-normal-learning.md)
+- [Can you give a visual explanation for the back propagation algorithm for neural networks?](./visual-backpropagation.md)
+- [Why did it take so long for deep networks to be invented?](./inventing-deeplearning.md)
+- [What are some good books/papers for learning deep learning?](./deep-learning-resources.md)
+- [Why are there so many deep learning libraries?](./many-deeplearning-libs.md)
+- [Why do some people hate neural networks/deep learning?](./deeplearning-criticism.md)
+- [How can I know if Deep Learning works better for a specific problem than SVM or random forest?](./deeplearn-vs-svm-randomforest.md)
+- [What is wrong when my neural network's error increases?](./neuralnet-error.md)
+- [How do I debug an artificial neural network algorithm?](./nnet-debugging-checklist.md)
+- [What is the difference between a Perceptron, Adaline, and neural network model?](./diff-perceptron-adaline-neuralnet.md)
+- [What is the basic idea behind the dropout technique?](./dropout.md)
+
+##### Other Algorithms for Supervised Learning
+
+- [Why is Nearest Neighbor a Lazy Algorithm?](./lazy-knn.md)
+
+##### Unsupervised Learning
+
+- [What are some of the issues with clustering?](./issues-with-clustering.md)
+
+##### Semi-Supervised Learning
+
+- [What are the advantages of semi-supervised learning over supervised and unsupervised learning?](./semi-vs-supervised.md)
+
+##### Ensemble Methods
+
+- [Is Combining Classifiers with Stacking Better than Selecting the Best One?](./logistic-boosting.md)
+
+##### Preprocessing, Feature Selection and Extraction
+
+- [Why do we need to re-use training parameters to transform test data?](./scale-training-test.md)
+- [What are the different dimensionality reduction methods in machine learning?](./dimensionality-reduction.md)
+- [What is the difference between LDA and PCA for dimensionality reduction?](./lda-vs-pca.md)
+- [When should I apply data normalization/standardization?](./when-to-standardize.md)
+- [Does mean centering or feature scaling affect a Principal Component Analysis?](./pca-scaling.md)
+- [How do you attack a machine learning problem with a large number of features?](./large-num-features.md)
+- [What are some common approaches for dealing with missing data?](./missing-data.md)
+- [What is the difference between filter, wrapper, and embedded methods for feature selection?](./feature_sele_categories.md)
+- [Should data preparation/pre-processing step be considered one part of feature engineering? Why or why not?](./dataprep-vs-dataengin.md)
+- [Is a bag of words feature representation for text classification considered as a sparse matrix?](./bag-of-words-sparsity.md)
+- [How can I apply an SVM to categorical data?](./svm_for_categorical_data.md)
+
+##### Naive Bayes
+
+- [Why is the Naive Bayes Classifier naive?](./naive-naive-bayes.md)
+- [What is the decision boundary for Naive Bayes?](./naive-bayes-boundary.md)
+- [Is it possible to mix different variable types in Naive Bayes, for example, binary and continues features?](./naive-bayes-vartypes.md)
+
+##### Other
+
+- [What is Euclidean distance in terms of machine learning?](./euclidean-distance.md)
+- [When should one use median, as opposed to the mean or average?](./median-vs-mean.md)
+
+##### Programming Languages and Libraries for Data Science and Machine Learning
+
+- [Is R used extensively today in data science?](./r-in-datascience.md)
+- [What is the main difference between TensorFlow and scikit-learn?](./tensorflow-vs-scikitlearn.md)
+
+<!--- end -->
+
+<br>
+<br>
+<br>
+
+
+
+
+### Questions about the Book
+
+- [Can I use paragraphs and images from the book in presentations or my blog?](./copyright.md)
+- [How is this different from other machine learning books?](./different.md)
+- [Which version of Python was used in the code examples?](./py2py3.md)
+- [Which technologies and libraries are being used?](./technologies.md)
+- [Which book version/format would you recommend?](./version.md)
+- [Why did you choose Python for machine learning?](./why-python.md)
+- [Why do you use so many leading and trailing underscores in the code examples?](./underscore-convention.md)
+- [What is the purpose of the `return self` idioms in your code examples?](./return_self_idiom.md)
+- [Are there any prerequisites and recommended pre-readings?](./prerequisites.md)
@@ -0,0 +1,13 @@
+# How are Artificial Intelligence and Machine Learning related?
+
+Artifical Intellicence (AI) started as a subfield of computer science with the focus on solving tasks that humans can but computers can't do (for instance, image recognition). AI can be approached in many ways, for example, writing a computer program that implements a set of rules devised by domain experts. Now, hand-crafting rules can be very laborious and time consuming.
+
+The field of machine learning -- originally, we can consider it as a subfield of AI -- was concerned with the development of algorithms so that computers can automatically learn (predictive) models from data.
+
+For instance, say we want to develop a program that can recognize handwritten digits from images. One would be to look at all of these images and come-up with a set of (nested) if-this-than-that rules to say which image is displayed in a particular image (for instance, by looking at the relative locations of pixels). Another approach would be to use a machine learning algorithm, which can fit a predictive model based on a thousands of labeled image samples that we may have collected in a database. Now, there's also deep learning, which in turn is a subfield of machine learning, referring to a particular subset of models that are particularly good at certain tasks such as image recognition and natural language processing.
+
+Or in short, machine learning (and deep learning) definitely helps to develop "AI," however, AI doesn't necessarily have to be developed using machine learning -- although, machine learning makes "AI" much more convenient ;).
+
+tldr; to summarize my point of view visually:
+
+![](ai-and-ml/ai-and-ml-1.png)
@@ -0,0 +1,13 @@
+# How can I avoid overfitting?
+
+In short, the general strategies are to
+
+1. collect more data
+2. use ensembling methods that "average" models
+3. choose simpler models / penalize complexity
+
+For the first point, it may help to plot learning curves, plotting the training vs. the validation or cross-validation performance. If you see a trend that more data helps with closing the cap between the two, and if you could afford collecting more data, then this would probably the best choice.
+
+In my experience, ensembling is probably the most convenient way to build robust predictive models on somewhat small-sized datasets. As in real life, consulting a bunch of "experts" is usually not a bad idea before making a decision ;).
+
+Regarding the third point, I usually start a predictive modeling task with the simplest model as a benchmark: usually logistic regression. Overfitting can be a real problem if our model has too much capacity — too many model parameters to fit, and too many hyperparameters to tune. If the dataset is small, a simple model is always a good option to prevent overfitting, and it is also a good benchmark for comparison to more "complex" alternatives.    
@@ -0,0 +1,41 @@
+# Is a bag of words feature representation for text classification considered as a sparse matrix?
+
+It depends on your vocabulary and dataset, but typically: Yes, definitely!
+
+
+By definition, a sparse matrix is called "sparse" if most of its elements are zero. In the bag of words model, each document is represented as a word-count vector. These counts can be binary counts (does a word occur or not) or absolute counts (term frequencies, or normalized counts), and the size of this vector is equal to the number of elements in your vocabulary. Thus, if most of your feature vectors are sparse, our bag-of-words feature matrix is most likely sparse as well!
+
+
+Now, the question is: "When are these feature vectors sparse?" It reallly depends on the size of our vocabulary, and the length and variety of the documents in our training corpus. For instance, the shorter and more similar the documents in our training set are, the more likely it is that we end up with a dense matrix, ---- it's still very unlikely in practice, though!
+
+Here's a trivial example ... Let's suppose we have 3 documents:
+
+
+- Doc1: Hello, World, the sun is shining
+- Doc2: Hello world, the weather is nice
+- Doc3: Hello world, the wind is cold
+
+
+Then, our vocabulary would look like this (using 1-grams without stop word removal):
+
+
+
+Vocabulary: [hello, world, the, wind, weather, sun, is, shining, nice, cold]
+
+
+The corresponding, binary feature vectors are:
+
+
+- Doc1: [1, 1, 1, 0, 0, 0, 1, 1, 0, 0]
+- Doc2: [1, 1, 1, 0, 0, 1, 0, 1, 1, 0]
+- Doc3: [1, 1, 1, 1, 0, 0, 1, 0, 0, 1]
+
+
+Which we use to construct the dense matrix:
+
+[[1, 1, 1, 0, 0, 0, 1, 1, 0, 0]
+[1, 1, 1, 0, 0, 1, 0, 1, 1, 0]
+[1, 1, 1, 1, 0, 0, 1, 0, 0, 1] ]
+
+
+As we can see, we have 17 x 1 and 13 x 0; so, by definition, this wouldn't be a sparse matrix. However, we can also guess how unlikely this scenario is in a real-world application ;)   
@@ -0,0 +1,46 @@
+# How does the random forest model work? How is it different from bagging and boosting in ensemble models?
+
+Let's assume we use a decision tree algorithm as base classifier for all three: boosting, bagging, and (obviously :)) the random forest.
+
+
+Why and when do we want to use any of these? Given a fixed-size number of training samples, our model will increasingly suffer from the "curse of dimensionality" if we increase the number of features. The challenge of individual, unpruned decision trees is that the hypothesis often ends up being too complex for the underlying training data -- decision trees are prone to overfitting.
+
+
+**tl;dr: Bagging and random forests are "bagging" algorithms that aim to reduce the complexity of models that overfit the training data. In contrast, boosting is an approach to increase the complexity of models that suffer from high bias, that is, models that underfit the training data.**
+
+
+## Bagging
+
+
+Now, let's take a look at the probably "simplest" case, bagging. Here, we train a number (ensemble) of decision trees from bootstrap samples of our training set. Bootstrap sampling means drawing random samples from our training set with replacement. E.g., if our training set consists of 7 training samples, our bootstrap samples (here: n=7) can look as follows, where C1, C2, ... Cm shall symbolize the decision tree classifiers:
+
+
+![](./bagging-boosting-rf/bagging.png)
+
+
+After we trained our (m) decision trees, we can use them to classify new data via majority rule. For instance, we'd let each decision tree make a decision and predict the class label that received more votes. Typically, this would result in a less complex decision boundary, and the bagging classifier would have a lower variance (less overfitting) than an individual decision tree. Below is a plot comparing a single decision tree (left) to a bagging classifier (right) for 2 variables from the Wine dataset (Alcohol and Hue).
+
+
+![](./bagging-boosting-rf/bagging-regions.png)
+
+
+## Boosting
+
+
+In contrast to bagging, we use very simple classifiers as base classifiers, so-called "weak learners." Picture these weak learners as "decision tree stumps" -- decision trees with only 1 splitting rule. Below, we will refer to the probably most popular example of boosting, AdaBoost. Here, we start with one decision tree stump (1) and "focus" on the samples it got wrong. In the next round, we train another decision tree stump that attempts to get these samples right (2); we achieve this by putting a larger weight on these training samples. Again, this 2nd classifier will likely get some other samples wrong, so we'd re-adjust the weights ...
+
+
+![](./bagging-boosting-rf/boosting.png)
+
+
+In a nutshell, we can summarize "Adaboost" as "adaptive" or "incremental"
+learning from mistakes. Eventually, we will come up with a model that has a lower bias than an individual decision tree (thus, it is less likely to underfit the training data).
+
+
+![](./bagging-boosting-rf/boosting-regions.png)
+
+
+## Random forests
+
+
+The random forest algorithm is actually a bagging algorithm: also here, we draw random bootstrap samples from our training set. However, in addition to the bootstrap samples, we also draw random subsets of features for training the individual trees; in bagging, we provide each tree with the full set of features. Due to the random feature selection, the trees are more independent of each other compared to regular bagging, which often results in better predictive performance (due to better variance-bias trade-offs), and I'd say that it's also faster than bagging, because each tree learns only from a subset of features.   
@@ -0,0 +1,17 @@
+# Which machine learning algorithms can be considered as among the best?
+
+I recommend taking a look at
+
+Wolpert, D.H., Macready, W.G. (1997), "[No Free Lunch Theorems for Optimization](http://ti.arc.nasa.gov/m/profile/dhw/papers/78.pdf)", IEEE Transactions on Evolutionary Computation 1, 67.
+
+Unfortunately, there's no real answer to this question: different datasets, questions, and assumptions require different algorithms -- or in other words: we haven't found the Master Algorithm, yet.
+
+But let me write down thoughts about different classifiers at least:
+
+- both logistic regression and SVMs work great for linear problems, logistic regression may be preferable for very noisy data
+- naive Bayes may work better than logistic regression for small training set sizes; the former is also pretty fast, e.g., if you have a large multi-class problem, you'd only have to train one classifier whereas you'd have to use One-vs-Rest or One-vs-One with in SVMs or logistic regression (alternatively, you could implement multinomial/softmax regression though); another point is that you don't have to worry so much about hyperparameter optimization -- if you are estimating the class priors from the training set, there are actually no hyperparameters
+- kernel SVM/logistic regression is preferable for nonlinear data vs. the linear models
+- k-nearest neighbor can also work quite well in practice for datasets with large number of samples and relatively low dimensionality
+- Random Forests & Extremely Randomized trees are very robust and work well across a whole range of problems -- linear and/or nonlinear problems
+
+Personally, I tend to prefer a multi-layer neural network in most cases, given that my dataset is sufficiently large. In my experience, the generalization performance was almost always superior to one of the other approaches I listed above. But again, it really depends on the given dataset.
@@ -0,0 +1,49 @@
+
+
+# What factors should I consider when choosing a predictive model technique?
+
+This is a very broad question, and the answer would basically fill an entire book. In a nutshell, I would come up with the
+
+### 1. How does your target variable look like?
+
+- continuous target variable? -> regression
+- categorical (nominal) target variable? -> classification
+- ordinal target variable? -> ranked classification
+- no target variable and want to find structure in data? -> cluster analysis, projection
+
+### 2. Is computational performance an issue?
+
+- use "cheaper" models/algorithms
+- dimensionality reduction
+- feature selection
+- lazy learner (e.g,. k-nearest neighbors)
+
+### 3. Does my dataset fit into memory? If no:
+
+- out of core learning
+- distributed systems
+
+### 4. Is my data linearly separable?
+
+- hard to know the answer upfront
+- always a good idea to compare different models
+
+### 5. Finding a good bias variance threshold. Does my model overfit?
+
+- increase regularization strength if supported by the model
+- dimensionality reduction or feature selection otherwise
+- collect more training data if possible (check via learning curves first)
+
+### 6. Are you planning to update your model with new data on the fly?
+
+- one option are lazy learners (e.g., K-nearest neighbors); needs to keep training data around; no learning necessary but more expensive predictions
+- it's generally relatively cheap to update generative models
+- another option is stochastic gradient descent for online learning
+
+...
+
+The list goes on and on :). I think Andreas Mueller's scikit-learn algorithm "cheat-sheet" is an excellent resource. (Click on the image to view the original, interactive version on scikit-learn)
+
+[![](./choosing-technique/scikit-cheatsheet.png)](http://scikit-learn.org/dev/tutorial/machine_learning_map/index.html)
+
+[Source: http://scikit-learn.org/dev/tutorial/machine_learning_map/index.html]
@@ -0,0 +1,62 @@
+# What are the broad categories of classifiers?
+
+
+### A (broad) categorization could be "discriminative" vs. "generative" classifiers:
+
+
+Discriminative algorithms:
+- a direct mapping of x -> y
+- intuition: "Distinguishing between people who are speaking different languages without actually learning the language"
+- e.g., Logistic regression, SVMs, Neural networks, ...
+
+Generative algorithms:
+- model how the data was generated (joint probability distributions p(x, y))
+- e.g., naive Bayes, Bayesian belief networks, Restricted Boltzmann machines
+
+
+### Or, we could categorize classifiers as "lazy" vs. "eager" learners:
+
+Lazy learners:
+- don't "learn" a decision rule (or function)
+- no learning step involved but require to keep training data around
+- e.g., K-nearest neighbor classifiers
+
+### A third possibility could be "parametric" vs. "non-parametric"
+
+(in context of machine learning; the field of statistics interprets use terms a little bit differently.)
+
+non-parametric:
+- representations grow with the training data size
+- e.g., Decision trees, K-nearest neighbors
+
+parametric:
+- representations are "fixed"
+- e.g., most linear classifiers like logistic regression etc.
+
+### Pedro Domingo's 5 Tribes of Machine Learning
+
+In his new book ([The Master Algorithm](http://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465065708/ref=sr_1_1?ie=UTF8&qid=1447045562&sr=8-1&keywords=pedro+domingos)), Pedro Domingo's mentioned the 5 tribes of machine learning, which is another nice categorization. Summarizing from the book (pp. 51-53)
+
+**Symbolists**  
+- manipulating symbols (like mathematicians replace expressions by expressions), or in other words, using pre-existing knowledge to fill in the missing pieces
+- "master algorithm:" inverse deduction
+
+
+**Connectionists**  
+- reverse-engineering a biological brain, i.e., strengthening the connections between neurons
+- "master algorithm:" backpropagation
+
+
+**Evolutionaries**
+- whereas connectionism is about fine-tuning the brain, evolution is about creating the brain
+- "master algorithm:" genetic programming
+
+**Bayesians**  
+- based on probabilistic inference, i.e., incorporating a priori knowledge: certain outcomes are more likely
+- "master algorithm:" Bayes' theorem and its derivatives
+
+**Analogizers**  
+- generalizing from similarity, i.e., recognizing similarities or in other words: remember experiences (training data) and how to combine them to make new predictions
+- "master algorithm:" support vector machine
+
+![](./classifier_categories/master_chart.jpg)
@@ -0,0 +1,46 @@
+# How was classification, as a learning machine, developed?
+
+
+There are two fundamental milestones I'd say.
+The first one is Fisher's Linear Discriminant [1], later generalized by Rao [2] to what we know as Linear Discriminant Analysis (LDA). Essentially, LDA is a linear transformation (or projection) technique, which is mainly used for dimensionality reduction (i.e., the objective is to find the k-dimensional feature subspace that -- linearly -- separates the samples from different classes best.
+Given the objective to maximize class separability, projecting the 2D dataset below onto "x-axis component," would be a better choice than the "y-axis component."
+
+
+
+![](./classifier-history/lda.png)
+
+Keep in mind though that LDA is a projection technique; the feature axes of your new feature subspace are (almost certainly) different from your original axes.
+In other words, LDA aims to find a new feature subspace that retains most of the class-discriminatory information. Where do I want to go with this? Intuitively, we can come up with a criterion function to minimize the ratio of the distance between the (class) sample means, and the within class scatter. And maximizing our criterion function plus plugging in a threshold function yields us a linear classifier.
+Anyway, subjectively, I would group LDA with its closed-form solution into the more classical statistics field (or probabilistic learning if you will due to its relation to ANOVA and Bayes' Theorem).
+
+
+Another timely take on classification would be the perceptron algorithm, which is in turn based on the concept of the McCulloch-Pitt (MCP) Neuron -- an early (maybe first?) model of how a neuron in a mammal's brain could work [3]. In contrast to the LDA classifier, Rosenblatt's perceptron [4] is an incremental learner. For each training sample, it compares the predicted class labes to the actual class label and modify the model weights accordingly.
+Rosenblatt's initial perceptron rule is fairly simple and can be summarized by the following steps:
+
+1. Initialize the weights to 0 or small random numbers.
+2. For each training sample x(i):
+- Calculate the output value.
+- Update the weights.
+The value for updating the weights at each increment is calculated by the learning rule
+
+![](./classifier-history/perceptron-rule.png)
+
+![](./classifier-history/perceptron-figure.png)
+
+There were few problems with this approach, i.e., the algorithm will never converge if the data is not perfectly separable by a line or hyperplane.
+An improvement over the perceptron was the adaptive linear neuron (Adaline) [5]. It is closely related to linear regression (if you use an optimization algorithm like gradient descent rather than the closed-form solution); we update the weights by comparing the real-value output to the actual class label. (Remember, the perceptron algorithms compares the binary-values class labels, i.e., the predicted and actual class labels.) After we trained the model, we use a threshold function to turn the real-valued output into a class label:
+
+![](./classifier-history/adaline.png)
+
+A very similar concept is Logistic Regression, however, instead of a linear function, we minimize a logistic function [6]. Again, it wouldn't say the early beginning of logistic regression would be necessarily the "machine learning" approach until incremental learning (gradient descent, stochastic gradient descent, and other optimization algorithms were being used to learn the model weights). In any case, scientists came up with many other cost functions until then, I'd say they are likely inspired by the perceptron, Adaline, and logistic regression.
+![](./classifier-history/activation-functions.png)
+
+For example, the hinge loss for SVM. Another direction of the analogizer's SVM approach would be the "connectionism," i.e., combining neuron units to multi-layer neural networks. Although scientists combined Adaline units to multiple Adalines (Madaline), the problem with Adaline was that a combination of linear units is ... well, it's still linear. In any case, there is so much to write about, but I hope that satisfies your curiosity towards the early beginning to some extend.
+
+
+[1]  Fisher, R. A. (1936). "The Use of Multiple Measurements in Taxonomic Problems". Annals of Eugenics 7 (2): 179–188. doi:10.1111/j.1469-1809.1936.tb02137.x  
+[2] Rao, R. C. (1948). "The utilization of multiple measurements in problems of biological classification". Journal of the Royal Statistical Society, Series B 10 (2): 159–203.  
+[3] McCulloch, W. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:115–133.  
+[4] F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.  
+[5] B. Widrow et al. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960.  
+[6] Berkson, Joseph. "Application of the logistic function to bio-assay." Journal of the American Statistical Association 39.227 (1944): 357-365.  
@@ -0,0 +1,20 @@
+# What are the best toy datasets to help visualize and understand classifier behavior?
+
+The visualization part is a bit tricky since we as humans are limited to 1-3 D graphics. However, I'd still say Iris is one of the most useful toy datasets for looking at classifier behavior (see image below).
+
+![](./clf-behavior-data/iris.png)
+
+(I've implemented this simple function here if you are interested: [mlxtend plot_decision_regions](http://rasbt.github.io/mlxtend/user_guide/plotting/plot_decision_regions/).)
+Other than that, I think that synthetic datasets like "XOR," "half-moons," or concentric circles would be good candidates for evaluating classifier on non-linear problems:
+
+<hr>
+
+![](./clf-behavior-data/xor.png)
+
+<hr>
+
+![](./clf-behavior-data/moons.png)
+
+<hr>
+
+![](./clf-behavior-data/circles.png)
@@ -0,0 +1,150 @@
+# Fitting a model via closed-form equations vs. Gradient Descent vs Stochastic Gradient Descent vs Mini-Batch Learning. What is the difference?
+
+
+In order to explain the differences between alternative approaches to estimating the parameters of a model, let's take a look at a concrete example: Ordinary Least Squares (OLS) Linear Regression.
+The illustration below shall serve as a quick reminder to recall the different components of a simple linear regression model:
+
+
+![](./closed-form-vs-gd/simple_regression.png)
+
+
+In Ordinary Least Squares (OLS) Linear Regression, our goal is to find the line (or hyperplane) that minimizes the vertical offsets. Or, in other words, we define the best-fitting line as the line that minimizes the sum of squared errors (SSE) or mean squared error (MSE) between our target variable (y) and our predicted output over all samples *i* in our dataset of size *n*.
+
+
+![](./closed-form-vs-gd/sse_mse.png)
+
+
+Now, we can implement a linear regression model for performing ordinary least squares regression using one of the following approaches:
+
+
+- Solving the model parameters analytically (closed-form equations)
+- Using an optimization algorithm (Gradient Descent, Stochastic Gradient Descent, Newton's Method, Simplex Method, etc.)
+
+
+### 1) Normal Equations (closed-form solution)
+
+
+The closed-form solution may (should) be preferred for "smaller" datasets -- if computing (a "costly") matrix inverse is not a concern. For very large datasets, or datasets where the inverse of **X**<sup>T</sup>**X** may not exist (the matrix is non-invertible or singular, e.g., in case of perfect multicollinearity), the GD or SGD approaches are to be preferred.
+The linear function (linear regression model) is defined as:
+
+
+![](./closed-form-vs-gd/linear_model.png)
+
+
+where *y* is the response variable, ***x*** is an *m*-dimensional sample vector, and ***w*** is the weight vector (vector of coefficients). Note that *w<sub>0</sub>* represents the y-axis intercept of the model and therefore *x<sub>0</sub>=1*.
+Using the closed-form solution (normal equation), we compute the weights of the model as follows:
+
+
+![](./closed-form-vs-gd/closed-form.png)
+
+### 2) Gradient Descent (GD)
+
+Using the Gradient Decent (GD) optimization algorithm, the weights are updated incrementally after each epoch (= pass over the training dataset).
+
+
+
+The cost function *J(&#8901;)*, the sum of squared errors (SSE), can be written as:
+
+
+![](./closed-form-vs-gd/j.png)
+
+
+The magnitude and direction of the weight update is computed by taking a step in the opposite direction of the cost gradient
+
+
+![](./closed-form-vs-gd/dw.png)
+
+where *&eta;* is the learning rate. The weights are then updated after each epoch via the following update rule:
+
+
+![](./closed-form-vs-gd/w_upd.png)
+
+
+where **&Delta;w** is a vector that contains the weight updates of each weight coefficient *w*, which are computed as follows:
+
+![](./closed-form-vs-gd/w_upd_expl.png)
+
+Essentially, we can picture GD optimization as a hiker (the weight coefficient) who wants to climb down a mountain (cost function) into a valley (cost minimum), and each step is determined by the steepness of the slope (gradient) and the leg length of the hiker (learning rate). Considering a cost function with only a single weight coefficient, we can illustrate this concept as follows:
+
+
+![](./closed-form-vs-gd/ball.png)
+
+
+### 3) Stochastic Gradient Descent (SGD)
+
+
+In GD optimization, we compute the cost gradient based on the complete training set; hence, we sometimes also call it *batch GD*. In case of very large datasets, using GD can be quite costly since we are only taking a single step for one pass over the training set -- thus, the larger the training set, the slower our algorithm updates the weights and the longer it may take until it converges to the global cost minimum (note that the SSE cost function is convex).
+
+In Stochastic Gradient Descent (SGD; sometimes also referred to as *iterative* or *on-line* GD), we **don't** accumulate the weight updates as we've seen above for GD:
+
+
+![](./closed-form-vs-gd/iter_gd.png)
+
+
+Instead, we update the weights after each training sample:
+
+
+![](./closed-form-vs-gd/iter_sgd.png)
+
+
+Here, the term "stochastic" comes from the fact that the gradient based on a single training sample is a "stochastic approximation" of the "true" cost gradient. Due to its stochastic nature, the path towards the global cost minimum is not "direct" as in GD, but may go "zig-zag" if we are visualizing the cost surface in a 2D space. However, it has been shown that SGD almost surely converges to the global cost minimum if the cost function is convex (or pseudo-convex)[1].
+Furthermore, there are different tricks to improve the GD-based learning, for example:
+
+
+- An adaptive learning rate &eta; Choosing a decrease constant *d* that shrinks the learning rate over time:
+![](./closed-form-vs-gd/adaptive_learning.png)
+
+
+- Momentum learning by adding a factor of the previous gradient to the weight update for faster updates:
+![](./closed-form-vs-gd/decrease_const.png)
+
+
+#### A note about shuffling
+
+
+There are several different flavors of SGD, which can be all seen throughout the literature. Let's take a look at the three most common variants:   
+
+
+
+
+
+
+
+##### A)
+
+- randomly shuffle samples in the training set
+    - for one or more epochs, or until approx. cost minimum is reached
+        - for training sample *i*
+            - compute gradients and perform weight updates
+
+##### B)
+
+- for one or more epochs, or until approx. cost minimum is reached
+    - randomly shuffle samples in the training set
+        - for training sample *i*
+            - compute gradients and perform weight updates
+
+##### C)
+
+- for iterations *t*, or until approx. cost minimum is reached:
+    - draw random sample from the training set
+        - compute gradients and perform weight updates
+
+
+In scenario A [3], we shuffle the training set only one time in the beginning; whereas in scenario B, we shuffle the training set after each epoch to prevent repeating update cycles. In both scenario A and scenario B, each training sample is only used once per epoch to update the model weights.
+
+
+In scenario C, we draw the training samples randomly with replacement from the training set [2]. If the number of iterations *t* is equal to the number of training samples, we learn the model based on a *bootstrap sample* of the training set.
+
+### 4) Mini-Batch Gradient Descent (MB-GD)
+
+Mini-Batch Gradient Descent (MB-GD) a compromise between batch GD and SGD. In MB-GD, we update the model based on smaller groups of training samples; instead of computing the gradient from 1 sample (SGD) or all *n* training samples (GD), we compute the gradient from *1 < k < n* training samples (a common mini-batch size is *k=50*).
+
+MB-GD converges in fewer iterations than GD because we update the weights more frequently; however, MB-GD let's us utilize vectorized operation, which typically results in a computational performance gain over SGD.
+
+
+### References
+
+- [1] Bottou, Léon (1998). "Online Algorithms and Stochastic Approximations". Online Learning and Neural Networks. Cambridge University Press. ISBN 978-0-521-65263-6
+- [2] Bottou, Léon. "Large-scale machine learning with SGD." Proceedings of COMPSTAT'2010. Physica-Verlag HD, 2010. 177-186.
+- [3] Bottou, Léon. "SGD tricks." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 421-436.
@@ -0,0 +1,120 @@
+This is an excerpt of an upcoming blog article of mine. Unfortunately, the blog article turned out to be quite lengthy, too lengthy. In the process of pruning, there are hard choices to be made, and this tangent, eh, section needs to go ...
+Before I hit the delete button ... maybe this section is useful to others!?
+
+Struggling to continue story where I left off: The "way" we select a model and select amongst different machine learning algorithms all depends on how we evaluate the different models, which in turn depends upon the performance metric we choose. To summarize, the topics we mostly care about are
+
+- estimation of the generalization performance
+- algorithm selection
+- hyperparameter tuning techniques
+- (cross)-validation and sampling techniques
+- performance metrics
+- class imbalances
+
+But now to the actual section I wanted to share ...
+
+## Interlude: Comparing and Computing Performance Metrics in Cross-Validation -- Imbalanced Class Problems and 3 Different Ways to Compute the F1 Score
+
+Not too long ago, George Forman and Martin Scholz wrote a thought-provoking paper dealing with the comparison and computation of performance metrics across literature, especially when dealing with class imbalances: [Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement (2010)](http://www.hpl.hp.com/techreports/2009/HPL-2009-359.pdf). This is such a nicely written, very accessible paper (and such an important topic)! I highly recommend given this a read. Given that it's not old hat to you, it might change your perspective, the way you read papers, the way you evaluate and benchmark your machine learning models -- and if you decide to publish your results, your readers will benefit as well, that's for sure.
+
+Now, imagine that we want to compare the performance of our new, shiny algorithm to the efforts made in the past. First, we want to make sure that we are comparing "fruits to fruits." Assuming we evaluate on the same dataset, we want to make sure that we use the same cross-validation technique and evaluation metric. I know, this sounds trivial, but we first want to establish this ground rule that we can't compare ROC areas under the curves (AUC) measures to F1 scores ...
+On a side note, the use of ROC AUC metrics is still a hot topic of discussion, e.g.,
+
+- JM. Lobo, A. Jiménez-Valverde, and R. Real 2008: [AUC: a misleading measure of the performance of predictive distribution models](http://onlinelibrary.wiley.com/doi/10.1111/j.1466-8238.2007.00358.x/abstract;jsessionid=40E65D14D4CEEC38F203699F5DCC18C7.f01t03?userIsAuthenticated=false&deniedAccessCustomisedMessage=)
+- Jin Huang & C. X. Ling 2005: [Using AUC and accuracy in evaluating learning algorithms](http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1388242&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D1388242)
+- AP. Bradley 1997 [The use of the area under the ROC curve in the evaluation of machine learning algorithms](http://www.sciencedirect.com/science/article/pii/S0031320396001422)
+
+In any case, let's focus on the F1 score for now summarizing some ideas from Forman & Scholz' paper after defining some of the relevant terminology.
+
+As we probably heard or read before, the F1-score is simply the harmonic mean of precision (PRE) and recall (REC)
+
+F1 = 2 * (PRE * REC) / (PRE + REC)
+
+***What we are trying to achieve with the F1-score metric is to find an equal balance between precision and recall, which is extremely useful in most scenarios when we are working with imbalanced datasets (i.e., a dataset with a non-uniform distribution of class labels).***
+
+---
+If we write the two metrics PRE and REC in terms of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), we get:
+
+- PRE = TP / (TP + FP)
+- REC = TP / (TP + FN)
+
+Thus, the precision score gives us an idea (expressed as a score from 1.0 to 0.0, from good to bad) of the proportion of how many actual spam emails (TP) we correctly classified as spam among all the emails we classified as spam (TP + FP).
+In contrast, the recall (also ranging from 1.0 to 0.0) tells us about how many of the actual spam emails (TP) we "retrieved" or "recalled" (TP + FN).
+
+---
+
+Okay, let's assume we settled on the F1-score as our performance metric of choice to benchmark our new algorithm; coincidentally, the algorithm in a certain paper, which should serve as our reference performance, was also evaluated using the F1 score. Using the same cross-validation technique on the same dataset, this should make this comparison, fair, right? No, no, no, not so fast! On top of choosing the appropriate performance metric -- comparing "fruits to fruits" -- we also have to care about how it's computed in order to compare "apples to apples." This is extremely important if we are comparing performance metrics on imbalanced datasets, which I will explain in a second (based on the results from Forman & Martin Scholz' paper). ***Also, keep in mind that even if our dataset doesn't seem to be imbalanced at first glance, let's think of the Iris dataset with 50 Setosa, 50 Virginica, and 50 Versicolor flowers: What happens if we use a One-vs-Rest (OVR; or One-vs-All, OVA) classification scheme?***
+
+In any case, let's focus on a binary classification problem (a *positive* and a *negative* class) for now using k-fold cross-validation as our cross-validation technique of choice for model selection.
+
+As mentioned before, we calculate the F1 score as
+
+F1 = 2 * (PRE * REC) / (PRE + REC)
+
+Now, what happens if we have a highly imbalanced dataset and perform our k-fold cross validation procedure in the training set? Well, chances are that a particular fold may not contain *a positive* sample so that TP=FN=0. If this doesn't sound too bad, have another look at the recall equation above -- yes, that's a zero-division error! Or, what happens if our classifier predicts the negative class almost all the time (i.e., it has a low false-positive rate)? Again, we get a zero-division error in the precision equation since TP = FP = 0.
+
+What can we do about it? There are two things. Firstly, let's stratify our folds -- stratification means that the random sampling procedure attempts to maintain the class-label proportion across the different folds. Thus, we are unlikely to face problems like having "no samples from the *positive* class" given that *k* is not larger than the number of *positive* samples in the training dataset.
+
+***In practice, different software packages handle the zero-division errors differently: Some don't hesitate throwing run-time exceptions; some may silently substitute the precision and/or recall by a 0 -- make sure what it's doing!*** On top of that, we can compute the F1 score in several distinct ways (and in multi-class problems, we can put the micro- and macro-averaging techniques on top of that, but this is beyond of the scope of this section). As listed by Forman and Scholz, these three different scenarios are
+
+#### (1)
+
+We compute the F1 score for each fold (iteration); then, we compute the average F1 score
+from these individual F1 scores.
+
+F1<sub>avg</sub> = 1/k &Sigma;<sup>k</sup><sub>i=1</sub> F1<sup>(i)</sup>
+
+
+#### (2)
+
+We compute the average precision and recall scores across the *k* folds; then, we use these average scores to compute the final F1 score.
+
+PRE = 1/k &Sigma;<sup>k</sup><sub>i=1</sub> PRE<sup>(i)</sup>
+
+REC = 1/k &Sigma;<sup>k</sup><sub>i=1</sub> REC<sup>(i)</sup>
+
+F1<sub>PRE, REC</sub> = 2 * (PRE * REC) / (PRE + REC)
+
+
+
+#### (3)
+
+We compute the number of TP, FP, and FN separately for each fold or iteration, and compute the final F1 score based on these "micro" metrics.
+
+TP = &Sigma;<sup>k</sup><sub>i=1</sub> TP<sup>(i)</sup>
+
+FP = &Sigma;<sup>k</sup><sub>i=1</sub> FP<sup>(i)</sup>
+
+FN = &Sigma;<sup>k</sup><sub>i=1</sub> FN<sup>(i)</sup>
+
+F1<sub>TP, FP, FN</sub> = (2 * TP) / (2 * TP + FP + FN)
+
+(Note that this equation doesn't suffer from the zero-division issue.)
+
+---
+
+Please note that we don't have to worry about the different ways to compute the classification error or accuracy, which we are working with in this blog article aside from this section. The reason is that it doesn't matter whether we we compute the accuracy as
+
+
+ACC<sub>avg</sub> = 1/k &Sigma;<sup>k</sup><sub>i=1</sub> ACC<sup>(i)</sup>
+
+or
+
+TP = &Sigma;<sup>k</sup><sub>i=1</sub> TP<sup>(i)</sup>
+
+TN = &Sigma;<sup>k</sup><sub>i=1</sub> TN<sup>(i)</sup>
+
+ACC <sub>avg</sub> = (TP + TN) / N
+
+The two approaches are identical, or with a more concrete example: (30 + 40) / 100  = (30/50 + 40/50) / 2 = 0.7.
+
+---
+
+Eventually, Forman and Scholz plaid this game of using different ways to compute the F1 score based on a benchmark dataset with a high-class imbalance (a bit exaggerated for demonstration purposes but not untypical when working with text data). It turns out that the resulting scores (from the identical model) differed substantially:
+
+- F1<sub>avg</sub>: 69%
+- F1<sub>PRE, REC</sub>: 73%
+- F1<sub>TP, FP, FN</sub>: 58%
+
+***Finally, based on further simulations, Forman and Scholz concluded that the computation of F1<sub>TP, FP, FN</sub> (compared to the alternative ways of computing the F1 score), were yielded the "most unbiased" estimate of the generalization performance using *k*-fold cross-validation.***
+
+In any case, the bottom line is that we should not only choose the appropriate performance metric and cross-validation technique for the task, but we also take a ***closer look at how the different performance metrics are computed in case we cite papers or rely on off-the-shelve machine learning libraries.***
@@ -0,0 +1,16 @@
+# Can I use paragraphs and images from the book in presentations or my blog?
+
+Probably, you know me as a big advocate of *open source* and I like to share useful resources wherever and whenever I can! Most of my projects really benefit from resources people shared with me, the community, and the Internet as a whole. I am more than happy to do so likewise, that's one of the reasons why Python is so great, but this is a topic for another story ... :)
+
+I am really passionate about machine learning, and other than using it for my own interests, I am truly excited about sharing knowledge, getting you excited about this field, and helping you on your path of becoming an affluent machine learning practitioner. It would make me really happy to hear if I was successful with my mission, and that you find the contents of my book are beneficial. If you are developing learning resources for other people, that's great, and if figures, code examples, math formulas, or passages are helpful for that, by all means, go for it!
+Unfortunately though, writing a book for most publishers comes with some restrictions, but I have spoken with the content manager of this book, who in turn talked to the *Copyright Team*.
+
+> I hope you don’t mind if ask you a few quick questions about the copyright for the images since I couldn’t find anything specific about the rules on the Packt website.
+For example, if someone wants to use one of the images from the book in
+a) a presentation for teaching purposes
+b) an article or blog post
+and agrees to cite the book and credit it properly, would this be okay with the copyright laws?
+
+Based on the response, I think it's quite positive overall. I want to post it verbatim to avoid any ambiguities that may arise to my reformulation of the exact words.
+
+> I have spoken to our copy rights team; they have confirmed we are fine for 3rd party users to use up to 10% of the book's content, as long as everything cited and credited properly. However, they are not allowed to redistribute or sell the information without our knowledge.   
@@ -0,0 +1,12 @@
+# What is the difference between a cost function and a loss function in machine learning?
+
+The terms *cost* and *loss* functions are synonymous (some people also call it error function). The more general scenario is to define an objective function first, which we want to optimize. This objective function could be to
+
+- maximize the posterior probabilities (e.g., naive Bayes)
+- maximize a fitness function (genetic programming)
+- maximize the total reward/value function (reinforcement learning)
+- maximize information gain/minimize child node impurities (CART decision tree classification)
+- minimize a mean squared error cost (or loss) function (CART, decision tree regression, linear regression, adaptive linear neurons, ...
+- maximize log-likelihood or minimize cross-entropy loss (or cost) function
+- minimize hinge loss (support vector machine)
+...
@@ -0,0 +1,7 @@
+# What learning path/discipline in data science I should focus on?
+
+The tl;dr: "Data science" is a broad field including many different specializations; which particular sub-role do you find most appealing? Statistics, machine learning, software engineering, data visualization...?
+
+Hm, how can I say this … it’s a pity that our day only has 24 hours, and there is only so much that we can learn as an individual person. So, I think it is somewhat important to reflect on your goals and your interest when you are picking your study topics. Although it has a “bad ring” to it, the phrase “being a jack of all trades” is certainly very tempting and useful if you work as an individual. As a “data scientist” there are endless topics you can get lost in, from programming to statistics, databases, differential calculus, … Of course, it’s useful to know a bit of everything, but I think it is impossible to become a master in everything in a given amount of time. For example, data scientists rarely work on their own but are often part of bigger teams with different areas of responsibilities. Some people are really good at stats; some people are responsible for building the framework to collect, clean, and extract data. Some people develop new algorithms, and some are “data scientist-programmers” who focus on implementing them most efficiently. For example, I’d say that I am a pretty good Python programmer, but my Java skills are really rudimentary. I think a better knowledge of Java would help me here and there, but my focus is more on the Machine Learning part; Python is already sufficient for me to implement all my ideas. So rather than investing in learning a brand new programming language, I try to focus more on my strength, e.g., picking up fresh ML concepts and staying on top with the current developments in the field. Of course, I’d like to learn a new programming language since it could be useful, but I also know that I only have so much time to learn all these things … ;)
+
+What I am trying to say is that it is probably not a bad idea to think about where you want to be in xx years, and what does it take to get there. Which are the things and skills that are necessary to get you there? I would focus on these first!
@@ -0,0 +1,21 @@
+# What are the different fields of study in data mining?
+
+
+I would roughly define the different application areas as
+
+1) Clustering (unsupervised learning)
+e.g., to find groups of customers based on some similarity
+
+2) Predictive modeling (supervised learning)
+2.1) Classification
+        e.g., medical diagnosis (sick/healthy), image classification etc.
+2.2) Regression
+      e.g., stock trade change prediction
+2.3) Ranking
+      e.g., search engine results
+
+3) Association rule mining
+e.g., which products do customers frequently buy together
+
+4) Anomaly detection
+e.g., credit fraud detection
@@ -0,0 +1,4 @@
+# What are differences in research nature between the two fields: Machine Learning &amp; Data Mining?
+
+In a nutshell, Data Mining is about the discovery of patterns in datasets or "gaining knowledge and insights" from data. Machine Learning is closely related though. We can think of Machine Learning algorithms as one of the work horses of Data Mining; most Data Mining approaches are based on Machine Learning algorithms. Maybe it helps to think of Data Mining as a pipeline of steps and approaches, and the use of a Machine Learning algorithm is one part of this pipeline.
+Or in other words, Data Mining is not "just" Machine Learning. E.g., data visualization or summarization is also part of Data Mining. What I was trying to say is that Machine Learning is one part, one set of techniques, that is/are being used in Data Mining.
@@ -0,0 +1,6 @@
+# Should data preparation/pre-processing step be considered one part of feature engineering? Why or why not?
+
+I think there's a fuzzy boundary between these two areas of tasks. I see data preparation more as a technical/computational task. E.g., if you think about getting the data into the "right" format, choosing the appropriate data structure / database, and so forth.
+Then, there's data cleaning, which can also be grouped into the "preparation / pre-processing" category. Here, you may want to think about detecting duplications, how to deal with outliers, and how to deal with missing data.
+
+To me, feature engineering is a bit different. I see it more as a "data/feature creation" step rather than a data "sanitizing" step. Feature engineering may include all different sorts of feature transformations in both directions: Higher-dimensional feature spaces (e.g., polynomials), lower dimensional feature spaces (dimensionality reduction like PCA, LDA, etc., hashing, clustering), or you keep the dimensions but change the distribution of your data (e.g., log transformation, standardization, min-max scaling etc.)
@@ -0,0 +1,30 @@
+# What are data science and machine learning?
+
+
+### Let's start with machine learning
+
+In short, machine learning algorithms are algorithms that learn (often predictive) models from data. I.e., instead of formulating "rules" manually, a machine learning algorithm will learn the model for you.
+
+![](./datascience-ml/ml-overview.jpg)
+
+So, let me give you an example to illustrate what that means! Say you are interested in implementing a spam filter. The probably most conservative approach would be to let a person sort these emails manually. Now, the "traditional" programming approach would be to look at some example emails (and/or use your "domain knowledge") to come up with a chain of rules like
+
+*"if this email contains word X, label it as spam, else if email contains ..."*
+
+Now,  machine learning algorithms help you formulating these rules. Or in other words, (supervised) machine learning algorithms will look at a dataset of labeled emails (spam and non-spam) and derive rules from there to separate the two classes.
+
+
+### So, what is data science then?
+
+First of all, "data science" is a pretty ambiguous, ill-defined term and interdisciplinary field; and people mean (expect) different things in different contexts. In my opinion, in practice, data science is pretty much the same as what we've known as *Data Mining* or *KDD* (Knowledge Discovery in Databases). The typical skills of a data scientists are
+
+- Computer science: programming, hardware understanding, etc.
+- Math: Linear algebra, calculus, statistics
+- Communication: visualization and presentation
+- Domain knowledge
+
+Where machine learning -- at its core -- is about the use and development of these learning algorithms, data science is more about the extraction of knowledge from data to answer particular question or solve particular problems.
+
+Machine learning is often a big part of a "data science" project, e.g., it is often heavily used for exploratory analysis and discovery (clustering algorithms) and building predictive models (supervised learning algorithms). However, in data science, you often also worry about the collection, wrangling, and cleaning of your data (i.e., data engineering), and eventually, you want to draw conclusions from your data that help you solve a particular problem.
+
+There are numerous examples of data science applications. Assume you are working for a credit company. Your boss gives you the task to find out whether a customer is creditworthy or not. You collect transaction data, maybe shipping records and customer ratings and so forth. Next, you'll probably use a machine learning algorithm to learn a predictive model. For example, let's assume you chose to grow a decision tree, and you concluded that this particular customer is not creditworthy. Finally, you prepare a nice presentation visualizing the decision tree to answer your boss' next question: Why is this customer not creditworthy? ...
@@ -0,0 +1,60 @@
+# Why are implementations of decision tree algorithms usually binary and what are the advantages of the different impurity metrics?
+
+For practical reasons (combinatorial explosion) most libraries implement decision trees with binary splits. The nice thing is that they are NP-complete (Hyafil, Laurent, and Ronald L. Rivest. "Constructing optimal binary decision trees is NP-complete." Information Processing Letters 5.1 (1976): 15-17.)
+
+Our objective function (e.g., in CART) is to maximize the information gain (IG) at each split:
+
+
+![](./decision-tree-binary/information-gain.png)
+
+
+where *f* is the feature to perform the split, and *D_p* and *D_j* are the datasets of the parent and *j*th child node, respectively. *I* is the impurity measure. *N* is the total number of samples, and *N_j* is the number of samples at the *j*th child node.
+Now, let's take a look at the most commonly used splitting criteria for classification (as described in CART). For simplicity, I will write the equations for the binary split, but of course it can be generalized for multiway splits. So, for a binary split we can compute *IG* as
+
+![](./decision-tree-binary/information-gain-2.png)
+
+Now, the two impurity measures or splitting criteria that are commonly used in binary decision trees are Gini Impurity(*I_G*) and Entropy (*I_H*) and the Classification Error (*I_E*). Let us start with the definition of Entropy, which is defined as
+
+![](./decision-tree-binary/entropy.png)
+
+for all “non-empty” classes
+
+![](./decision-tree-binary/empty-classes.png)
+
+and *p(i|t)* is the proportion of the samples that belong to class *c* for a particular node *t*. The entropy is therefore 0 if all samples at a node belong to the same class, and the Entropy is maximal if we have an uniform class distribution
+Intuitively, the Gini Impurity can be understood as a criterion to minimize the probability of misclassification
+
+![](./decision-tree-binary/gini-impurity.png)
+
+Similar to the Entropy, the Gini Impurity is maximal if the classes are perfectly mixed.
+However, in practice both Gini Impurity and Entropy typically yield very similar results and it is often not worth spending much time on evaluating trees using different impurity criteria rather than experimenting with different pruning cut-offs.
+Another impurity measure is the Classification Error
+
+![](./decision-tree-binary/error.png)
+
+which is a useful criterion for pruning but not recommend for growing a decision tree since it is less sensitive to changes in the class probabilities of the nodes.
+
+![](./decision-tree-binary/overview-plot.png)
+
+So let me illustrate what I mean by "the classification error is less sensitive to changes in the class probabilities" by looking at the two possible splitting scenarios shown in the figure below.
+
+![](./decision-tree-binary/split.png)
+
+We start with a data set *D_p* at the parent node that consists 40 samples from class 1 and 40 samples from class 2 that we split into two datasets D_left and D_right, respectively. The information gain using the Classification Error as splitting criterion would be the same (*IG_E* = 0.25) in both scenario *A* and *B*:
+
+![](./decision-tree-binary/calc_1.png)
+
+![](./decision-tree-binary/calc_2.png)
+
+However, the Gini Impurity would favor the split in scenario B (0.1666) over scenario A (0.125), which is indeed more “pure”:
+
+![](./decision-tree-binary/calc_3.png)
+
+Similarly, the entropy criterion would favor scenario B(IGH = 0.31) over scenario A(IGH = 0.19):
+
+
+![](./decision-tree-binary/calc_5.png)
+
+![](./decision-tree-binary/calc_6.png)
+
+Maybe some more words about Gini vs. Entropy. As mentioned before, the resulting trees are typically very similar in practice. Maybe an advantage of Gini would be that you don't need to compute the log, which can make it a bit faster in your implementation.  
@@ -0,0 +1,19 @@
+# What are the disadvantages of using classic decision tree algorithm for a large dataset?
+
+
+
+### The computational efficiency perspective
+
+It's a combinatorial search problem: at each split, we want to find the features that give us "the best bang for the buck" (maximizing information gain). If we choose a"brute" force approach, our computational complexity is O(m^2), where m is the number of features in our training set, and O(n^2) for the number of n training cases (I think it can be O(n log(n) if you are lucky).
+
+Let's take a look at a simple dataset, Iris (150 flowers, 3 classes, 4 continuous features). At each split, we have to re-evaluate all 4 features, and for each feature we have to find the optimal value to split on, e.g,. sepal length <3.4 cm (this is for a binary split). Computational complexity is one of the reasons why people implement *binary* decision trees most of the time.
+
+### The predictive performance perspective
+
+An unpruned model is much more likely to overfit as a consequence of the curse of dimensionality. However, instead of pruning a single decision tree, it often a better idea to use ensemble methods. We could
+
+- combine decision tree stumps that learn from each other by focusing on samples that are hard to classify (AdaBoost)
+- create an ensemble of unpruned decision trees; draw bootstrap samples, and do random feature selection (random forests)
+- forget about bagging and use all training samples as input for your unpruned trees; choose both the splitting feature and splitting value at random (= Extremely randomized trees)   
+
+(Related topic: [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](../bagging-boosting-rf.md))
@@ -0,0 +1,75 @@
+# Why are we growing decision trees via entropy instead of the classification error?
+
+Before we get to the main question -- the real interesting part -- let's take a look at some of the (classification) decision tree basics to make sure that we are on the same page.
+
+##### The Basic Algorithm
+
+1. Start at the root node as parent node
+2. Split the parent node at the feature *x<sub>i</sub>* to minimize the sum of the child node impurities (maximize information gain)
+3. Assign training samples to new child nodes
+4. Stop if leave nodes are pure or early stopping criteria is satisfied, else repeat steps 1 and 2 for each new child node
+
+##### Stopping Rules
+
+1. The leaf nodes are pure
+2. A maximal node depth is reached
+3. Splitting a note does not lead to an information gain*
+
+\* This is the important part as we will see later.
+
+
+##### Impurity Metrics and Information Gain
+
+Formally, we can write the "Information Gain" as
+
+![](./decisiontree-error-vs-entropy/ig.png)
+
+![](./decisiontree-error-vs-entropy/ig_xp.png)
+
+(Note that since the parent impurity is a constant, we could also simply compute the average child node impurities, which would have the same effect.)
+For simplicity, we will only compare the "Entropy" criterion to the classification error; however, the same concepts apply to the Gini index as well.
+
+We write the Entropy equation as
+
+![](./decisiontree-error-vs-entropy/entropy_eq.png)
+
+
+for all non-empty classed *p(i | t)* &ne; 0, where *p(i | t)* is the proportion (or frequency or probability) of the samples that belong to class *i* for a particular node *t*; *C* is the number of unique class labels.
+
+![](./decisiontree-error-vs-entropy/entropy_plot.png)
+
+
+
+
+Although we are all very familiar with the classification error, we write it down for completeness:
+
+![](./decisiontree-error-vs-entropy/error_eq.png)
+
+![](./decisiontree-error-vs-entropy/error_plot.png)
+
+
+
+### Classification Error vs. Entropy
+
+
+Here comes the more interesting part, just as promised. Let's consider the following binary tree starting with a training set of 40 "positive" training samples (y=1) and 80 training samples from the "negative" class (y=0). Further, let's assume that it is possible to come up with 3 splitting criteria (based on 3 binary features x1, x2, and x3) that can separate the training samples perfectly:
+
+![](./decisiontree-error-vs-entropy/Slide1.png)
+
+Now, is it possible to learn this hypothesis (i.e., tree model) by minimizing the classification error as a criterion function? Let's do the math:
+
+![](./decisiontree-error-vs-entropy/Slide2.png)
+
+As we can see, the Information Gain after the first split is exactly 0, since average classification error of the 2 child nodes is exactly the same as the classification error of the parent node (40/120 = 0.3333333). In this case, splitting the initial training set wouldn't yield any improvement in terms of our classification error criterion, and thus, the tree algorithm would stop at this point (for this statement to be true, we have to make the assumption that neither splitting on feature x2 nor x3 would lead to an Information gain as well).
+
+Next, let's see what happens if we use Entropy as an impurity metric:
+
+![](./decisiontree-error-vs-entropy/Slide3.png)
+
+In contrast to the average classification error, the average child node entropy is **not** equal to the entropy of the parent node. Thus, the splitting rule would continue until the child nodes are pure (after the next 2 splits). So, why is this happening? For an intuitive explanation, let's zoom in into the Entropy plot:
+
+![](./decisiontree-error-vs-entropy/entropy_annotated.png)
+
+The green square-shapes are the Entropy values for p(28/70) and (12/50) of the first two child nodes in the decision tree model above, connected by a green (dashed) line. To recapitulate: the decision tree algorithm aims to find the feature and splitting value that leads to a maximum decrease of the average child node impurities over the parent node.
+So, if we have 2 entropy values (left and right child node), the average will fall onto the straight, connecting line.
+However **-- and this is the important part --** we can see that the Entropy is always larger than the averaged Entropy due to its "bell shape," which is why we keep continuing to split the nodes in contrast to the classification error.
@@ -0,0 +1,11 @@
+# What are some good books/papers for learning deep learning?
+
+
+A good overview and introduction is the recent deep learning review published in Nature (http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html); it references a lot of useful literature to follow up on
+
+- LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature521.7553 (2015): 436-444.
+
+
+As a good textbook resource, I would like to recommend Yoshua Bengio's upcoming "Deep Learning" book. The book is freely accessible at: http://goodfeli.github.io/dlbook/
+
+- Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. "Deep learning." An MIT Press book in preparation. Draft chapters available at http://www. iro. umontreal. ca/∼ bengioy/dlbook (2014).
@@ -0,0 +1,32 @@
+# How can I know if Deep Learning works better for a specific problem than SVM or random forest?
+
+If we tackle a supervised learning problem, my advice is to start with the simplest hypothesis space first. I.e., try a linear model such as logistic regression. If this doesn't work "well" (i.e., it doesn't meet our expectation or performance criterion that we defined earlier), I would move on to the next experiment.
+
+
+### Random Forests vs. SVMs
+
+I would say that random forests are probably THE "worry-free" approach - if such a thing exists in ML: There are no real hyperparameters to tune (maybe except for the number of trees; typically, the more trees we have the better). On the contrary, there are a lot of knobs to be turned in SVMs: Choosing the "right" kernel, regularization penalties, the slack variable, ...
+
+Both random forests and SVMs are non-parametric models (i.e., the complexity grows as the number of training samples increases). Training a non-parametric model can thus be more expensive, computationally, compared to a generalized linear model, for example. The more trees we have, the more expensive it is to build a random forest. Also, we can end up with a lot of support vectors in SVMs; in the worst-case scenario, we have as many support vectors as we have samples in the training set. Although, there are multi-class SVMs, the typical implementation for mult-class classification is One-vs.-All; thus, we have to train an SVM for each class -- in contrast, decision trees or random forests, which can handle multiple classes out of the box.
+
+To summarize, random forests are much simpler to train for a practitioner; it's easier to find a good, robust model. The complexity of a random forest grows with the number of trees in the forest, and the number of training samples we have. In SVMs, we typically need to do a fair amount of parameter tuning, and in addition to that, the computational cost grows linearly with the number of classes as well.
+
+### Deep Learning
+
+As a rule of thumb, I'd say that SVMs are great for relatively small data sets with fewer outliers. Random forests may require more data but they almost always come up with a pretty robust model. And deep learning algorithms... well, they require "relatively" large datasets to work well, and you also need the infrastructure to train them in reasonable time. Also, deep learning algorithms require much more experience: Setting up a neural network using deep learning algorithms is much more tedious than using an off-the-shelf classifiers such as random forests and SVMs.
+On the other hand, deep learning really shines when it comes to complex problems such as image classification, natural language processing, and speech recognition. Another advantage is that you have to worry less about the feature engineering part. Again, in practice, the decision which classifier to choose really depends on your dataset and the general complexity of the problem -- that's where your experience as machine learning practitioner kicks in.
+
+If it comes to predictive performance, there are cases where SVMs do better than random forests and vice versa:
+
+- Caruana, Rich, and Alexandru Niculescu-Mizil. "[An empirical comparison of supervised learning algorithms.](https://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml06.pdf)" Proceedings of the 23rd international conference on Machine learning. ACM, 2006.
+
+The same is true for deep learning algorithms if you look at the MNIST benchmarks ([http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)):
+The best-performing model in this set is a committee consisting of 35 ConvNets, which were reported to have a 0.23% test error; the best SVM model has a test error of 0.56%. The ConvNet ensemble may reach a better accuracy (for the sake of this ensemble, let's pretend that these are totally unbiased estimates), but without a question, I'd say that the 35 ConvNet committee is far more expensive (computationally). So, if you make that decision: Is a 0.33% improvement worth it? In some cases, it's maybe worth it (e.g., in the financial sector for non-real time predictions), in other cases it perhaps won't be worth it, though.
+
+
+
+So, my practical advice is:
+- Define a performance metric to evaluate your model
+- Ask yourself: What performance score is desired, what hardware is required, what is the project deadline
+- Start with the simplest model
+- If you don't meet your expected goal, try more complex models (if possible)
@@ -0,0 +1,10 @@
+# Why do some people hate neural networks/deep learning?
+
+I also know many people who make disrespectful remarks about neural networks in genaral. Personally, I find recurrent and convolutional neural networks truly beautiful. However, there is this popular saying: "that if al that you have is a hammer everything starts to look like a nail"
+
+The math behind neural nets is probably a bit harder to understand, but I don't think they are really black boxes. I think a neural net is not more of a black box than standard techniques like kernel SVMs of random forests. Actually, I think it is easier to explain backpropagation than kernel methods.
+
+However, I think people in the biosciences prefer "interpretable" results, e.g., decision trees where they can follow the "reasoning" step by step. Unarguably, random forests are better at solving the prediction task since you don't have to worry so much about overfitting or pruning your tree; at the same time, we lose some of this "interpretability." Although, I think this is not really true. Feature importance computed from e.g., extremely randomized trees might be even more useful than looking at a single decision tree.
+
+Please note that I don't blame the bio-research field for this kind of thinking, they really try to solve different problems.
+Assuming biologists want to know which functional groups of a ligand are "interacting" with residues in the protein binding site. Of course, the primary goal is often to a good agonist or antagonist (inhibitor or drug) in a million-compound database to solve a particular problem; in addition, they are also trying to "understand" and "explain" the results.
@@ -0,0 +1,26 @@
+# What is the Definition of Data Science?
+
+That's a tricky question. It's particularly challenging because there is no "definition." Unfortunately, this rather novel term has been used ambiguously, and there are many subjective definitions. Anyway, I think the field of Data Science is highly interdisciplinary and influenced by many many other fields ...
+
+Okay, before I start, I’d say data science is mainly about extracting knowledge from data (and I think that the terms “data mining” or “Knowledge Discovery in Databases” are highly related). So, data science can be about analyzing trends, building predictive models, … etc.
+
+It’s say data science is an agglomerate of  data collection, data modeling and analysis, decision making, and everything you need to know to accomplish your goals with respect to the aforementioned sub-tasks. Eventually, it boils down to the following fields/skills:
+
+There’s computer science:
+
+- algorithms
+- programming (patterns, languages etc.),
+- understanding hardware & operating systems
+- high-performance computing'
+
+And the mathematical aspects:
+
+- linear algebra
+- differential equations for optimization problems
+- statistics
+
+Plus others:
+
+- machine learning of course (somewhere between the technical and mathematical skills)
+- domain knowledge
+- and very important: the data visualization & communication skills
@@ -0,0 +1,120 @@
+# What is the difference between a Perceptron, Adaline, and neural network model?
+
+Both Adaline and the Perceptron are (single-layer) neural network models.
+The Perceptron is one of the oldest and simplest learning algorithms out there, and I would consider Adaline as an improvement over the Perceptron.
+
+
+### What Adaline and the Perceptron have in common
+
+- they are classifiers for binary classification
+- both have a linear decision boundary
+- both can learn iteratively, sample by sample (the Perceptron naturally, and Adaline via stochastic gradient descent)
+- both use a threshold function
+
+Before we talk about the differences, let's talk about the inputs first. The first step in the two algorithms is to compute the so-called net input *z* as the linear combination of our feature variables *x* and the model weights *w*.
+
+![](./diff-perceptron-adaline-neuralnet/1.png)
+
+
+Then, in the Perceptron and Adaline, we define a threshold function to make a prediction. I.e., if *z* is greater than a threshold theta, we predict class 1, and 0 otherwise:
+
+![](./diff-perceptron-adaline-neuralnet/2.png)
+
+![](./diff-perceptron-adaline-neuralnet/3.png)
+
+### The differences between the Perceptron and Adaline
+
+- the Perceptron uses the class labels to learn model coefficients
+- Adaline uses continuous predicted values (from the net input) to learn the model coefficients, which is more "powerful" since it tells us by "how much" we were right or wrong
+
+So, in the perceptron, as illustrated below, we simply  use the predicted class labels to update the weights, and in Adaline, we use a continuous response:
+
+![](./diff-perceptron-adaline-neuralnet/4.png)
+
+(Note that I inserted the "activation function" in Adaline just for illustrative purposes; here, this activation function is simply the identity function)
+Both learning algorithms can actually be summarized by 4 simple steps -- given that we use stochastic gradient descent for Adaline:
+
+1. Initialize the weights to 0 or small random numbers.
+2. For each training sample:
+    1. Calculate the output value.
+    2. Update the weights.
+
+We write the weight update in each iteration as:
+
+![](./diff-perceptron-adaline-neuralnet/5.png)
+
+where
+
+![](./diff-perceptron-adaline-neuralnet/6.png)
+
+
+Again, the "output" is the continuous net input value in Adaline and the predicted class label in case of the perceptron; eta is the learning rate.
+(In case you are interested: This weight update in Adaline is basically just taking the "opposite step" in direction of the sum-of-squared error cost gradient. I've a more detailed walkthrough [here](http://rasbt.github.io/mlxtend/user_guide/general_concepts/linear-gradient-derivative/) on deriving the cost gradient.
+
+
+### Multi-layer neural networks
+
+Although you haven't asked about multi-layer neural networks specifically, let me add a few sentences about one of the oldest and most popular multi-layer neural network architectures: the Multi-Layer Perceptron (MLP). The term "Perceptron" is a little bit unfortunate in this context, since it really doesn't have much to do with Rosenblatt's Perceptron algorithm.
+
+![](./diff-perceptron-adaline-neuralnet/7.png)
+
+MLPs can basically be understood as a network of multiple artificial neurons over multiple layers. Here, the activation function is not linear (like in Adaline), but we use a non-linear activation function like the logistic sigmoid (the one that we use in logistic regression) or the hyperbolic tangent, or a piecewise-linear activation function such as the rectifier linear unit (ReLU). In addition, we often use a softmax function (a generalization of the logistic sigmoid for multi-class problems) in the output layer, and a threshold function to turn the predicted probabilities (by the softmax) into class labels.
+
+
+So, what the advantage of the MLP over the classic Perceptron and Adaline? By connecting the artificial neurons in this network through non-linear activation functions, we can create complex, non-linear decision boundaries that allow us to tackle problems where the different classes are not linearly separable.
+
+Let me show you an example :)
+
+
+![](./diff-perceptron-adaline-neuralnet/8.png)
+
+Here's the Python code if you want to reproduce these plots:
+
+```Python
+from mlxtend.plotting import plot_decision_regions
+from mlxtend.classifier import Perceptron
+from mlxtend.classifier import Adaline
+from mlxtend.classifier import MultiLayerPerceptron
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.datasets import make_moons
+import matplotlib.gridspec as gridspec
+import itertools
+
+gs = gridspec.GridSpec(2, 2)xw
+X, y = make_moons(n_samples=100, random_state=123)
+fig = plt.figure(figsize=(10,8))
+
+ppn = Perceptron(epochs=50, eta=0.05, random_seed=0)
+ppn.fit(X, y)
+ada = Adaline(epochs=50, eta=0.05, random_seed=0)
+ada.fit(X, y)
+
+mlp = MultiLayerPerceptron(n_output=len(np.unique(y)),
+                           n_features=X.shape[1],
+                           n_hidden=150,
+                           l2=0.0,
+                           l1=0.0,
+                           epochs=500,
+                           eta=0.01,
+                           alpha=0.0,
+                           decrease_const=0.0,
+                           minibatches=1,
+                           shuffle_init=False,
+                           shuffle_epoch=False,
+                           random_seed=0)
+
+mlp = mlp.fit(X, y)
+
+
+for clf, lab, grd in zip([ppn, ppn, mlp],
+                         ['Perceptron', 'Adaline', 'MLP (logistic sigmoid)'],
+                         itertools.product([0, 1], repeat=2)):
+
+    clf.fit(X, y)
+    ax = plt.subplot(gs[grd[0], grd[1]])
+    fig = plot_decision_regions(X=X, y=y, clf=clf, legend=2)
+    plt.title(lab)
+
+plt.show()
+```
@@ -0,0 +1,39 @@
+# What is the difference between deep learning and usual machine learning?
+
+That's an interesting question, and I try to answer this in a very general way.
+
+In essence, deep learning offers a set of techniques and algorithms that help us to parameterize deep neural network structures -- artificial neural networks with many hidden layers and parameters.
+One of the key ideas behind deep learning is to extract high level features from the given dataset. Thereby, deep learning aims to overcome the challenge of the often tedious feature engineering task and helps with parameterizing traditional neural networks with many layers.
+
+
+Now, to introduce deep learning, let us take a look at a more concrete example involving multi-layer perceptrons (MLPs).
+
+
+On a tangent: The term "perceptron" in MLPs may be a bit confusing since we don't really want only linear neurons in our network. Using MLPs, we want to learn complex functions to solve non-linear problems. Thus, our network is conventionally composed of one or multiple "hidden" layers that connect the input and output layer. Those hidden layers normally have some sort of sigmoid activation function (log-sigmoid or the hyperbolic tangent etc.). For example, think of a log-sigmoid unit in our network as a logistic regression unit that returns continuous values outputs in the range 0-1. A simple MLP could look like this
+
+![](./difference-deep-and-normal-learning/mlp.png)
+
+
+where y_hat is the final class label that we return as the prediction based on the inputs (x) if this are classification tasks. The "a"s are our activated neurons and the "w"s are the weight coefficients.
+Now, if we add multiple hidden layers to this MLP, we'd also call the network "deep." The problem with such "deep" networks is that it becomes tougher and tougher to learn "good" weights for this network. When we start training our network, we typically assign random values as initial weights, which can be terribly off from the "optimal" solution we want to find. During training, we then use the popular backpropagation algorithm (think of it as reverse-mode auto-differentiation) to propagate the "errors" from right to left and calculate the partial derivatives with respect to each weight to take a step into the opposite direction of the cost (or "error") gradient. **Now, the problem with deep neural networks is the so-called "vanishing gradient" -- the more layers we add, the harder it becomes to "update" our weights because the signal becomes weaker and weaker. Since our network's weights can be terribly off in the beginning (random initialization) it can become almost impossible to parameterize a "deep" neural network with backpropagation.**
+
+**Deep Learning**
+
+Now, this is where "deep learning" comes into play. Roughly speaking, we can think of deep learning as "clever" tricks or algorithms that can help us with the training of such "deep" neural network structures. There are many, many different neural network architectures, but to continue with the example of the MLP, let me introduce the idea of convolutional neural networks (ConvNets). We can think of those as an "add-on" to our MLP that helps us to detect features as "good" inputs for our MLP.
+
+In applications of "usual" machine learning, there is typically a strong focus on the feature engineering part; the model learned by an algorithm can only be so good as its input data. Of course, there must be sufficient discriminatory information in our dataset, however, the performance of machine learning algorithms can suffer substantially when the information is buried in meaningless features. The goal behind deep learning is to automatically learn the features from (somewhat) noisy data; it's about algorithms that do the feature engineering for us to provide deep neural network structures with meaningful information so that it can learn more effectively. **We can think of deep learning as algorithms for automatic "feature engineering," or we could simply call them "feature detectors," which help us to overcome the vanishing gradient challenge and facilitate the learning in neural networks with many layers.**
+
+
+
+Let's consider a ConvNet in context of image classification.
+Here, we use so-called "receptive fields" (think of them as "windows") that slide over our image. We then connect those "receptive fields" (for example of the size of 5x5 pixel) with 1 unit in the next layer, this is the so-called "feature map." After this mapping, we have constructed a so-called convolutional layer. Note that our feature detectors are basically replicates of one another -- they share the same weights. The idea is that if a feature detector is useful in one part of the image it is likely that it is useful somewhere else, but at the same time it allows each patch of image to be represented in several ways.
+
+![](./difference-deep-and-normal-learning/convolution.png)
+
+
+Next, we have a "pooling" layer, where we reduce neighboring features from our feature map into single units (by taking the max feature or by averaging them, for example). We do this over many rounds and eventually arrive at an almost scale invariant representation of our image (the exact term is "equivariant"). This is very powerful since we can detect objects in an image no matter where they are located.
+
+
+![](./difference-deep-and-normal-learning/convnet.png)   
+
+In essence, the "convolutional" add-on that acts as a feature extractor or filter to our MLP. Via the convolutional layers we aim to extract the useful features from the images, and via the pooling layers, we aim to make the features somewhat equivariant to scale and translation.
@@ -0,0 +1,17 @@
+# What is the difference between a classifier and a model?
+
+Essentially, the terms "classifier" and "model" are synonymous in certain contexts; however, sometimes people refer to "classifier" as the learning algorithm that learns the model from the training data. To makes things more tractable, let's define some of the key terminology:
+
+- ***Training sample:*** A training sample is a data point *x* in an available training set that we use for tackling a predictive modeling task. For example, if we are interested in classifying emails, one email in our dataset would be one training sample. Sometimes, people also use the synonymous terms *training instance* or *training example*.
+
+- ***Target function:*** In predictive modeling, we are typically interested in modeling a particular process; we want to learn or approximate a particular function that, for example, let's us distinguish spam from non-spam email. The ***target function*** *f(x) = y* is the **true** function *f* that we want to model.
+
+- ***Hypothesis:*** A hypothesis is a certain function that we believe (or hope) is similar to the true function, the *target function* that we want to model. In context of email spam classification, it would be the *rule* we came up with that allows us to separate spam from non-spam emails.
+
+- ***Model:*** In machine learning field, the terms *hypothesis* and *model* are often used interchangeably. In other sciences, they can have different meanings, i.e., the hypothesis would be the "educated guess" by the scientist, and the *model* would be the manifestation of this *guess* that can be used to test the hypothesis.
+
+- ***Learning algorithm:*** Again, our goal is to find or approximate the ***target function***, and the learning algorithm is a set of instructions that tries to *model* the target function using our training dataset. A learning algorithm comes with a ***hypothesis space***, the set of possible hypotheses it can come up with in order to model the unknown target function by formulating the *final hypothesis*
+
+- ***Classifier:*** A classifier is a special case of a *hypothesis* (nowadays, often learned by a machine learning algorithm). A *classifier* is a *hypothesis* or *discrete-valued function* that is used to assign (categorical) class labels to particular data points. In the email classification example, this classifier could be a hypothesis for labeling emails as spam or non-spam. However, a *hypothesis* must not necessarily be synonymous to a *classifier*. In a different application, our *hypothesis* could be a function for mapping study time and educational backgrounds of students to their future SAT scores.
+
+So, we can say that a *classifier* is a special case of a *hypothesis* or *model*: a classifier is a function that assigns a class label to a data point.
@@ -0,0 +1,15 @@
+# How is this different from other machine learning books?
+
+Good question! There are already a bunch of books out there that explain how to do machine learning using Python. So, when the publisher approached me and asked if I would be interested in writing "Python Machine Learning," my first thought was: "Does the world really need another book about machine learning and Python?" I gave myself a week to think about it, and meanwhile I browsed through (most of) the other books that were out there. I am afraid to say so, but I honestly didn't like them much. The reason is that they largely treat machine learning as a black box, and most of them read like a scikit-learn documentation.
+Also, I didn't want to write another theoretical book since there are already many excellent ones out there, for example, C.M. Bishop's "Pattern Recognition and Machine Learning" or "The Elements of Statistical Learning" by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. My take on this topic would be a mix between the two extremes, and I wanted to introduce the machine learning theory but provide practical examples as well. Those "code examples" are supposed to 1) illustrate the concepts to aid the learning experience, and 2) serve as a reference so that readers can directly apply those techniques for real-world problem solving.
+
+<hr>
+
+**The tl:dr; version:**
+
+This is not yet just another "this is how scikit-learn works" book. I aim to explain how Machine Learning works, tell you everything you need to know in terms of best practices and caveats, and then we will learn how to put those concepts into action using NumPy, scikit-learn, Theano and so on :).
+
+<hr>
+
+Sure, this book will also contain a decent amount of "math & equations," and in my opinion, there is no way around it if we want to get away from the "black box thinking." However, I hope I managed to make it really easy to follow so that it can be read by a person who doesn't have a strong math background. Many parts of this book will provide examples in scikit-learn, in my opinion the most beautiful and practical machine learning library. However, we will also implement certain algorithms, which are not part of scikit-learn yet, by ourselves to boost our learning experience, for example, adaptive linear neurons, sequential feature selection algorithms, and multilayer artificial neural networks with backpropagation.
+However, this book is not only about learning algorithms, but I also have a strong emphasis on "best practices" and everything that comes before and after the "model learning" in a machine learning application pipeline. Dealing with missing data, transforming categorical data into proper formats, extracting features, evaluating models via cross-validation, comparing algorithms with nested cross-validation, plotting receiver operator characteristics just to name a few ... All in all, this (350 page) short book provides you with everything you need to apply machine learning to real-world problem solving!
@@ -0,0 +1,31 @@
+# What are the different dimensionality reduction methods in machine learning?
+
+Since there are so many different approaches, let's break it down to "feature selection" and "feature extraction."
+
+Some examples of feature selection:
+
+- L1 regularization (e.g., Logistic regression) and sparsity
+- variance thresholds
+- recursive feature elimination based on the weights of linear models
+- random forests / extra trees and feature importance (calculated as average information gain)
+- sequential forward/backward selection
+- genetic algorithms
+- exhaustive search
+
+Some examples of feature extraction:
+
+- Principal Component Analysis (PCA), unsupervised, returns axes of maximal variance given the constraint that those axes are orthogonal to each other
+- Linear Discriminant Analysis (LDA; not to be confused with Latent Dirichlett Allocation), supervised, returns axes that maximizes class separability (same constraint that axes are also orthogonal); and another article: Linear Discriminant Analysis bit by bit
+- kernel PCA: uses kernel trick to transform non-linear data to a feature space were samples may be linearly separable (in contrast, LDA and PCA are linear transformation techniques
+- supervised PCA
+- and many more non-linear transformation techniques, which you can find nicely summarized here: [Nonlinear dimensionality reduction](https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction)
+
+** So, which technique should we use? **
+
+This also follows the "No Lunch Theorem" principle in some sense: there is no method that is always superior; it depends on your dataset. Intuitively, LDA would make more sense than PCA if you have a linear classification task, but empirical studies showed that it is not always the case. Although kernel PCA can separate concentric circles, it fails to unfold the Swiss Rroll, for example; here, locally linear embedding (LLE) would be more appropriate.  
+
+![](./dimensionality-reduction/swiss-roll.png)
+
+![](./dimensionality-reduction/rbf-kpca.png)
+
+![](./dimensionality-reduction/lle.png)
--- a/Show More
+++ b/Show More