Fast Approximate Spectral Normalization for Robust Deep Neural Networks
This work addresses the need for efficient and robust DNNs in safety-critical applications, representing an incremental improvement over existing spectral normalization techniques.
The paper tackles the problem of making deep neural networks robust to adversarial attacks by introducing a fast approximate spectral normalization algorithm, which improves time efficiency by up to 60% and model robustness by 61% on average compared to state-of-the-art methods.
Deep neural networks (DNNs) play an important role in machine learning due to its outstanding performance compared to other alternatives. However, DNNs are not suitable for safety-critical applications since DNNs can be easily fooled by well-crafted adversarial examples. One promising strategy to counter adversarial attacks is to utilize spectral normalization, which ensures that the trained model has low sensitivity towards the disturbance of input samples. Unfortunately, this strategy requires exact computation of spectral norm, which is computation intensive and impractical for large-scale networks. In this paper, we introduce an approximate algorithm for spectral normalization based on Fourier transform and layer separation. The primary contribution of our work is to effectively combine the sparsity of weight matrix and decomposability of convolution layers. Extensive experimental evaluation demonstrates that our framework is able to significantly improve both time efficiency (up to 60\%) and model robustness (61\% on average) compared with the state-of-the-art spectral normalization.