The Singular Values of Convolutional Layers
This work addresses regularization challenges in deep learning for computer vision, offering an incremental improvement over existing methods.
The paper tackles the problem of characterizing and efficiently computing the singular values of convolutional layers in neural networks, resulting in a new regularization method that improves test error on CIFAR-10 from 6.2% to 5.3%.
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2\% to 5.3\%.