Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimum
This provides theoretical guarantees for global convergence in neural network training, addressing optimization challenges for researchers in machine learning theory.
The authors analyzed single-layer neural networks with Xavier initialization in the asymptotic limit of large hidden units and training steps, proving convergence to a random ODE with a Gaussian distribution that minimizes a convex quadratic objective, leading to zero training loss under certain assumptions.
We analyze single-layer neural networks with the Xavier initialization in the asymptotic regime of large numbers of hidden units and large numbers of stochastic gradient descent training steps. The evolution of the neural network during training can be viewed as a stochastic system and, using techniques from stochastic analysis, we prove the neural network converges in distribution to a random ODE with a Gaussian distribution. The limit is completely different than in the typical mean-field results for neural networks due to the $\frac{1}{\sqrt{N}}$ normalization factor in the Xavier initialization (versus the $\frac{1}{N}$ factor in the typical mean-field framework). Although the pre-limit problem of optimizing a neural network is non-convex (and therefore the neural network may converge to a local minimum), the limit equation minimizes a (quadratic) convex objective function and therefore converges to a global minimum. Furthermore, under reasonable assumptions, the matrix in the limiting quadratic objective function is positive definite and thus the neural network (in the limit) will converge to a global minimum with zero loss on the training set.