Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
This provides theoretical guarantees for training neural networks in the over-parameterized regime, which is incremental but addresses a key bottleneck in deep learning optimization.
The paper tackles the optimization landscape of over-parameterized shallow neural networks, showing that with quadratic activations, globally optimal models can be found efficiently for arbitrary data, and for differentiable activations, gradient descent converges linearly to global optima under Gaussian inputs and planted weights.
In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization landscape of training such shallow neural networks has certain favorable characteristics that allow globally optimal models to be found efficiently using a variety of local search heuristics. This result holds for an arbitrary training data of input/output pairs. For differentiable activation functions we also show that gradient descent, when suitably initialized, converges at a linear rate to a globally optimal model. This result focuses on a realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted weight coefficients.