Tunable Sensitivity to Large Errors in Neural Network Training
This addresses the challenge of handling large errors in training for neural network practitioners, though it appears incremental as it modifies an existing gradient-based approach.
The paper tackles the problem of neural network training by proposing a tunable sensitivity to hard examples, inspired by human learning, which generalizes the cross-entropy gradient step. The method reduces test prediction error compared to vanilla cross-entropy on benchmark datasets, with optimal sensitivity correlated to network depth.
When humans learn a new concept, they might ignore examples that they cannot make sense of at first, and only later focus on such examples, when they are more useful for learning. We propose incorporating this idea of tunable sensitivity for hard examples in neural network learning, using a new generalization of the cross-entropy gradient step, which can be used in place of the gradient in any gradient-based training method. The generalized gradient is parameterized by a value that controls the sensitivity of the training process to harder training examples. We tested our method on several benchmark datasets. We propose, and corroborate in our experiments, that the optimal level of sensitivity to hard example is positively correlated with the depth of the network. Moreover, the test prediction error obtained by our method is generally lower than that of the vanilla cross-entropy gradient learner. We therefore conclude that tunable sensitivity can be helpful for neural network learning.