Global Convergence of SGD On Two Layer Neural Nets
This provides theoretical guarantees for training neural networks, addressing a fundamental problem in machine learning, though it is incremental as it builds on prior analysis of Villani functions.
The paper proves global convergence of stochastic gradient descent (SGD) for two-layer neural networks with smooth activations like sigmoid and tanh, under specific initializations, and shows an exponential convergence rate for continuous-time SGD, with regularization independent of network size.
In this note, we consider appropriately regularized $\ell_2-$empirical risk of depth $2$ nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it -- for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.