wehub-resource-sync
18 lines
1.9 KiB
Markdown
18 lines
1.9 KiB
Markdown
# How do I debug an artificial neural network algorithm?
|
|
|
|
There are many, many reasons that can explain a unexpected, "bad" performance of neural networks. Let's compile a quick check list that we can process in a somewhat sequential manner to get to the root of that problem
|
|
|
|
1. Is the data set okay? More concretely: Is there a lot of noise? Are the features "powerful" enough to discriminate between classes? (It's a good idea to try a bunch of off-the-shelf classifiers to get an initial benchmark; classifiers like random forest, softmax regression, or kernel SVM)
|
|
2. Did we forget standardizing the features?
|
|
3. Did we implement and use gradient checking to make sure that our implementation is correct?
|
|
4. Do we use a random weight initialization scheme (e.g., from a random normal distribution multiplied by a small coefficient < 0) vs. initializing the model parameters to all-zero weights?
|
|
5. Did we try to increase or decrease the learning rate?
|
|
6. Have we checked that the cost decreases over time? If yes, have we tried to increase the number of epochs?
|
|
7. Have we tried to modify the learning rate using momentum learning and/or a decrease constant (e.g., AdaGrad)
|
|
8. Have we tried different non-linear activation functions other than the one we are currently using (e.g., logistic sigmoid, tanh, or ReLU)?
|
|
9. When you estimated the performance of our network via cross-validation (for example, via holdout or k-fold), did we notice a large discrepancy between training and validation performance? A substantial difference in performance on training and validation sets may indicate that we are overfitting the training data too much. As a remedy, we could try to
|
|
1. collect more training samples if possible
|
|
2. decrease the complexity of your network (e.g,. fewer nodes, fewer hidden layers)
|
|
3. implement dropout
|
|
4. add a penalty against complexity to the cost function (e.g., L2 regularization)
|