Global Convergence of Second-order Dynamics in Two-layer Neural Networks
This work addresses a theoretical gap in optimization for neural networks by extending convergence guarantees to momentum-based methods, which is incremental but important for understanding training dynamics in deep learning.
The paper tackles the problem of whether second-order dynamics with momentum, specifically the heavy ball method, guarantee global convergence to a global optimum in two-layer neural networks, similar to known results for first-order gradient flow. They prove positive convergence results asymptotically in the mean field limit, with numerical simulations suggesting this holds for reasonably small networks.
Recent results have shown that for two-layer fully connected neural networks, gradient flow converges to a global optimum in the infinite width limit, by making a connection between the mean field dynamics and the Wasserstein gradient flow. These results were derived for first-order gradient flow, and a natural question is whether second-order dynamics, i.e., dynamics with momentum, exhibit a similar guarantee. We show that the answer is positive for the heavy ball method. In this case, the resulting integro-PDE is a nonlinear kinetic Fokker Planck equation, and unlike the first-order case, it has no apparent connection with the Wasserstein gradient flow. Instead, we study the variations of a Lyapunov functional along the solution trajectories to characterize the stationary points and to prove convergence. While our results are asymptotic in the mean field limit, numerical simulations indicate that global convergence may already occur for reasonably small networks.