Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
This provides a theoretical guarantee for training deep networks with less over-parameterization, which is incremental but useful for researchers in optimization and deep learning.
The paper tackles the problem of proving global convergence for deep neural networks by showing that a single wide layer of width N after the input, with constant-width pyramidal layers, suffices for gradient descent to find a global minimum, reducing over-parameterization requirements compared to prior work.
Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with $N$ ($N$ being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width $N$ following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used LeCun's initialization and obtain an over-parameterization requirement for the single wide layer of order $N^2.$