Preprint: Norm Loss: An efficient yet effective regularization method for deep neural networks
This addresses training stability and performance problems for deep learning practitioners, but it is incremental as it builds on existing regularization and normalization techniques.
The authors tackled issues like exploding gradients and covariant-shift in deep neural networks by proposing Norm Loss, a weight soft-regularization method based on the Oblique manifold, which pushes weight vectors to have a norm close to one. Results on CIFAR-10, CIFAR-100, and ImageNet datasets with ResNet architectures show it is competitive or superior to state-of-the-art methods with negligible computational overhead and less sensitivity to hyperparameters.
Convolutional neural network training can suffer from diverse issues like exploding or vanishing gradients, scaling-based weight space symmetry and covariant-shift. In order to address these issues, researchers develop weight regularization methods and activation normalization methods. In this work we propose a weight soft-regularization method based on the Oblique manifold. The proposed method uses a loss function which pushes each weight vector to have a norm close to one, i.e. the weight matrix is smoothly steered toward the so-called Oblique manifold. We evaluate our method on the very popular CIFAR-10, CIFAR-100 and ImageNet 2012 datasets using two state-of-the-art architectures, namely the ResNet and wide-ResNet. Our method introduces negligible computational overhead and the results show that it is competitive to the state-of-the-art and in some cases superior to it. Additionally, the results are less sensitive to hyperparameter settings such as batch size and regularization factor.