Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers
This work addresses a bottleneck in training robust CNNs by providing a more efficient and tighter bound, though it is incremental as it builds on existing spectral norm methods.
The paper tackled the problem of accurately bounding the spectral norm of convolutional layers in CNNs to enhance generalization and stability, resulting in a new upper bound that is independent of input resolution and efficiently computable, with experiments showing improved performance.
Controlling the spectral norm of the Jacobian matrix, which is related to the convolution operation, has been shown to improve generalization, training stability and robustness in CNNs. Existing methods for computing the norm either tend to overestimate it or their performance may deteriorate quickly with increasing the input and kernel sizes. In this paper, we demonstrate that the tensor version of the spectral norm of a four-dimensional convolution kernel, up to a constant factor, serves as an upper bound for the spectral norm of the Jacobian matrix associated with the convolution operation. This new upper bound is independent of the input image resolution, differentiable and can be efficiently calculated during training. Through experiments, we demonstrate how this new bound can be used to improve the performance of convolutional architectures.