Revisiting Spatial Invariance with Low-Rank Local Connectivity
This work addresses the design of inductive biases in deep learning for computer vision, offering an incremental improvement by suggesting spatial invariance may be overly restrictive.
The authors tackled the problem of whether relaxing spatial invariance in convolutional neural networks could improve performance, finding that their low-rank locally connected layers increased classification accuracy over both convolutional and locally connected layers on MNIST, CIFAR-10, and CelebA datasets.
Convolutional neural networks are among the most successful architectures in deep learning with this success at least partially attributable to the efficacy of spatial invariance as an inductive bias. Locally connected layers, which differ from convolutional layers only in their lack of spatial invariance, usually perform poorly in practice. However, these observations still leave open the possibility that some degree of relaxation of spatial invariance may yield a better inductive bias than either convolution or local connectivity. To test this hypothesis, we design a method to relax the spatial invariance of a network layer in a controlled manner; we create a \textit{low-rank} locally connected layer, where the filter bank applied at each position is constructed as a linear combination of basis set of filter banks with spatially varying combining weights. By varying the number of basis filter banks, we can control the degree of relaxation of spatial invariance. In experiments with small convolutional networks, we find that relaxing spatial invariance improves classification accuracy over both convolution and locally connected layers across MNIST, CIFAR-10, and CelebA datasets, thus suggesting that spatial invariance may be an overly restrictive prior.