wehub-resource-sync
24 lines
1.9 KiB
Markdown
24 lines
1.9 KiB
Markdown
# When should I apply data normalization/standardization?
|
|
|
|
|
|
The only family of algorithms that I could think of being scale-invariant are tree-based methods. Let's take the general CART decision tree algorithm. Without going into much depth regarding information gain and impurity measures, we can think of the decision as "is feature x_i >= some_val?" Intuitively, we can see that it really doesn't matter on which scale this feature is (centimeters, Fahrenheit, a standardized scale -- it really doesn't matter).
|
|
|
|
|
|
Some examples of algorithms where feature scaling matters are:
|
|
|
|
|
|
- k-nearest neighbors with an Euclidean distance measure if want all features to contribute equally
|
|
- k-means (see k-nearest neighbors)
|
|
- logistic regression, SVMs, perceptrons, neural networks etc. if you are using gradient descent/ascent-based optimization, otherwise some weights will update much faster than others
|
|
- linear discriminant analysis, principal component analysis, kernel principal component analysis since you want to find directions of maximizing the variance (under the constraints that those directions/eigenvectors/principal components are orthogonal); you want to have features on the same scale since you'd emphasize variables on "larger measurement scales" more.
|
|
|
|
|
|
There are many more cases than I can possibly list here ... I always recommend you to think about the algorithm and what it's doing, and then it typically becomes obvious whether we want to scale your features or not.
|
|
|
|
|
|
In addition, we'd also want to think about whether we want to "standardize" or "normalize" (here: scaling to [0, 1] range) our data. Some algorithms assume that our data is centered at 0. For example, if we initialize the weights of a small multi-layer perceptron with tanh activation units to 0 or small random values centered around zero, we want to update the model weights "equally."
|
|
As a rule of thumb I'd say: When in doubt, just standardize the data, it shouldn't hurt.
|
|
|
|
|
|
|