Analytical Moment Regularizer for Gaussian Robust Networks
This addresses the issue of noise robustness in deep learning for vision tasks, offering an efficient alternative to data augmentation, though it is incremental in nature.
The authors tackled the problem of deep neural networks' sensitivity to noise by proposing a new training regularizer that minimizes the expected loss under Gaussian input noise, resulting in a robustness boost equivalent to 3-21 folds of data augmentation.
Despite the impressive performance of deep neural networks (DNNs) on numerous vision tasks, they still exhibit yet-to-understand uncouth behaviours. One puzzling behaviour is the subtle sensitive reaction of DNNs to various noise attacks. Such a nuisance has strengthened the line of research around developing and training noise-robust networks. In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input. We provide an efficient and simple approach to approximate such a regularizer for arbitrary deep networks. This is done by leveraging the analytic expression of the output mean of a shallow neural network; avoiding the need for the memory and computationally expensive data augmentation. We conduct extensive experiments on LeNet and AlexNet on various datasets including MNIST, CIFAR10, and CIFAR100 demonstrating the effectiveness of our proposed regularizer. In particular, we show that networks that are trained with the proposed regularizer benefit from a boost in robustness equivalent to performing 3-21 folds of data augmentation.