Noise Injection Node Regularization for Robust Learning
This addresses robustness issues for users of deep neural networks, but it is incremental as it builds on existing regularization techniques.
The paper tackles the problem of improving robustness in deep neural networks against test data perturbations by introducing Noise Injection Node Regularization (NINR), which injects structured noise during training, resulting in substantial improvements, such as outperforming methods like Dropout or L2 regularization in some cases for unstructured noise.
We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initialization conditions such that noise is the dominant driver of dynamics at the start of training. As it simply requires the addition of external nodes without altering the existing network structure or optimization algorithms, this method can be easily incorporated into many standard problem specifications. We find improved stability against a number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms other existing methods such as Dropout or $L_2$ regularization, in some cases. We further show that desirable generalization properties on clean data are generally maintained.