Equivalence of approximation by convolutional neural networks and fully-connected networks
This work addresses a foundational problem in machine learning theory by bridging the analysis of CNNs and FCNs, providing a theoretical equivalence that could influence future research in neural network approximation theory.
The paper tackles the theoretical gap between convolutional neural networks (CNNs) and fully-connected networks (FCNs) by proving that approximation bounds for FCNs translate to similar bounds for CNNs on translation equivariant functions, specifically for CNNs without pooling and with circular convolutions.
Convolutional neural networks are the most widely used type of neural networks in applications. In mathematical analysis, however, mostly fully-connected networks are studied. In this paper, we establish a connection between both network architectures. Using this connection, we show that all upper and lower bounds concerning approximation rates of {fully-connected} neural networks for functions $f \in \mathcal{C}$ -- for an arbitrary function class $\mathcal{C}$ -- translate to essentially the same bounds concerning approximation rates of convolutional neural networks for functions $f \in {\mathcal{C}^{equi}}$, with the class ${\mathcal{C}^{equi}}$ consisting of all translation equivariant functions whose first coordinate belongs to $\mathcal{C}$. All presented results consider exclusively the case of convolutional neural networks without any pooling operation and with circular convolutions, i.e., not based on zero-padding.