Covariance in Physics and Convolutional Neural Networks
This provides a theoretical foundation for CNN design, but it is incremental as it builds on existing ideas without introducing new methods or data.
The paper examines the concept of covariance in convolutional neural networks (CNNs) by comparing it to its use in theoretical physics, and shows that assuming covariance along with locality, linearity, and weight sharing uniquely determines the convolution form.
In this proceeding we give an overview of the idea of covariance (or equivariance) featured in the recent development of convolutional neural networks (CNNs). We study the similarities and differences between the use of covariance in theoretical physics and in the CNN context. Additionally, we demonstrate that the simple assumption of covariance, together with the required properties of locality, linearity and weight sharing, is sufficient to uniquely determine the form of the convolution.