Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
This work provides a theoretical explanation for a key empirical observation in deep learning, addressing a foundational issue for researchers and practitioners.
The authors tackled the problem of why larger neural networks generalize better by theoretically analyzing a 3-layer convolutional network in an XOR problem extension, showing that overparameterization leads to better generalization due to weight clustering and feature exploration at initialization, with empirical validation on MNIST.
Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization. However, there is currently no theoretical analysis that explains this observation. In this work, we provide theoretical and empirical evidence that, in certain cases, overparameterized convolutional networks generalize better than small networks because of an interplay between weight clustering and feature exploration at initialization. We demonstrate this theoretically for a 3-layer convolutional neural network with max-pooling, in a novel setting which extends the XOR problem. We show that this interplay implies that with overparamterization, gradient descent converges to global minima with better generalization performance compared to global minima of small networks. Empirically, we demonstrate these phenomena for a 3-layer convolutional neural network in the MNIST task.