ASNI: Adaptive Structured Noise Injection for shallow and deep neural networks
This work addresses regularization for neural network training, offering a method that may enhance performance without extra computational cost, but it appears incremental as it builds directly on dropout.
The authors tackled the problem of improving neural network regularization by proposing Adaptive Structured Noise Injection (ASNI), a generalization of dropout where noise is not independent but follows a joint distribution, and empirically showed it boosts accuracy, disentangles representations, and leads to sparser representations in feedforward and convolutional networks.
Dropout is a regularisation technique in neural network training where unit activations are randomly set to zero with a given probability \emph{independently}. In this work, we propose a generalisation of dropout and other multiplicative noise injection schemes for shallow and deep neural networks, where the random noise applied to different units is not independent but follows a joint distribution that is either fixed or estimated during training. We provide theoretical insights on why such adaptive structured noise injection (ASNI) may be relevant, and empirically confirm that it helps boost the accuracy of simple feedforward and convolutional neural networks, disentangles the hidden layer representations, and leads to sparser representations. Our proposed method is a straightforward modification of the classical dropout and does not require additional computational overhead.