Spectral Norm Regularization for Improving the Generalizability of Deep Learning
This work addresses the generalizability issue in deep learning for practitioners, but it is incremental as it builds on existing regularization techniques.
The authors tackled the problem of deep learning models' poor generalizability due to high sensitivity to input perturbations by proposing spectral norm regularization, which penalizes the spectral norm of weight matrices, and experimentally showed it improves generalizability over baseline methods.
We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spectral norm regularization, which penalizes the high spectral norm of weight matrices in neural networks. We provide supportive evidence for the abovementioned hypothesis by experimentally confirming that the models trained using spectral norm regularization exhibit better generalizability than other baseline methods.