Exact Spectral Norm Regularization for Neural Networks
This work addresses the need for more effective regularization methods in deep learning to enhance model robustness and generalization, though it is incremental as it builds on prior spectral regularization approaches.
The authors tackled the problem of regularizing the spectral norm of neural network Jacobians by developing a scheme that targets the exact spectral norm, achieving improved generalization performance and strong protection against natural and adversarial noise compared to previous techniques.
We pursue a line of research that seeks to regularize the spectral norm of the Jacobian of the input-output mapping for deep neural networks. While previous work rely on upper bounding techniques, we provide a scheme that targets the exact spectral norm. We showcase that our algorithm achieves an improved generalization performance compared to previous spectral regularization techniques while simultaneously maintaining a strong safeguard against natural and adversarial noise. Moreover, we further explore some previous reasoning concerning the strong adversarial protection that Jacobian regularization provides and show that it can be misleading.