Asymptotic Singular Value Distribution of Linear Convolutional Layers
This work addresses computational bottlenecks in analyzing and regularizing convolutional neural networks, offering incremental improvements for researchers and practitioners in deep learning.
The paper tackled the problem of approximating singular values in linear convolutional layers, which are computationally expensive to compute directly, by developing an asymptotic spectral density method that improved accuracy over existing circular approximations and provided effective spectral norm upper bounds for regularization.
In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a "wrapping around" operation can transform linear convolution to a circular one, by which the singular values can be approximated with reduced computational complexity by those of a block matrix with doubly circulant blocks, the accuracy of such an approximation is not guaranteed. In this paper, we propose to inspect such a linear transformation matrix through its asymptotic spectral representation - the spectral density matrix - by which we develop a simple singular value approximation method with improved accuracy over the circular approximation, as well as upper bounds for spectral norm with reduced computational complexity. Compared with the circular approximation, we obtain moderate improvement with a subtle adjustment of the singular value distribution. We also demonstrate that the spectral norm upper bounds are effective spectral regularizers for improving generalization performance in ResNets.