Spectral Norm of Convolutional Layers with Circular and Zero Paddings
This work addresses the need for efficient and robust Lipschitz layers in neural networks, particularly for enhancing adversarial robustness, though it appears incremental by generalizing an existing method to new padding types.
The paper tackles the problem of computing the spectral norm of convolutional layers with circular and zero paddings using Gram iteration, proving quadratic convergence and bridging gaps between padding types, and demonstrates that their spectral rescaling method outperforms state-of-the-art techniques in precision, computational cost, and scalability.
This paper leverages the use of \emph{Gram iteration} an efficient, deterministic, and differentiable method for computing spectral norm with an upper bound guarantee. Designed for circular convolutional layers, we generalize the use of the Gram iteration to zero padding convolutional layers and prove its quadratic convergence. We also provide theorems for bridging the gap between circular and zero padding convolution's spectral norm. We design a \emph{spectral rescaling} that can be used as a competitive $1$-Lipschitz layer that enhances network robustness. Demonstrated through experiments, our method outperforms state-of-the-art techniques in precision, computational cost, and scalability. The code of experiments is available at https://github.com/blaisedelattre/lip4conv.