The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization
This work provides a systematic theoretical explanation for the superior generalization of convolutional networks over fully connected baselines, which is a foundational problem for understanding deep learning architectures.
This paper investigates how the inductive biases of convolutional neural networks (CNNs), specifically locality and weight sharing, alter the implicit regularization from the edge-of-stability phenomenon in gradient descent. It demonstrates that CNNs generalize on spherical data with a rate of n^(-1/6 + O(m/d)), a scenario where fully connected networks fail, by coupling learned filters to the low-dimensional patch manifold.
We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent. Prior work has established that for fully connected networks, the strength of this regularization is governed solely by the global input geometry; consequently, it is insufficient to prevent overfitting on difficult distributions such as the high-dimensional sphere. In this paper, we show that locality and weight sharing fundamentally change this picture. Specifically, we prove that provided the receptive field size $m$ remains small relative to the ambient dimension $d$, these networks generalize on spherical data with a rate of $n^{-\frac{1}{6} +O(m/d)}$, a regime where fully connected networks provably fail. This theoretical result confirms that weight sharing couples the learned filters to the low-dimensional patch manifold, thereby bypassing the high dimensionality of the ambient space. We further corroborate our theory by analyzing the patch geometry of natural images, showing that standard convolutional designs induce patch distributions that are highly amenable to this stability mechanism, thus providing a systematic explanation for the superior generalization of convolutional networks over fully connected baselines.