On the Convergence of SGD Training of Neural Networks
This work addresses a foundational issue in machine learning by questioning widely held assumptions about optimization dynamics, potentially influencing training strategies for neural networks.
The paper challenges the common belief that local minima and valleys significantly affect SGD optimization in neural networks, proposing instead that SGD's behavior is better described as the simultaneous convergence of many largely non-interacting subproblems.
Neural networks are usually trained by some form of stochastic gradient descent (SGD)). A number of strategies are in common use intended to improve SGD optimization, such as learning rate schedules, momentum, and batching. These are motivated by ideas about the occurrence of local minima at different scales, valleys, and other phenomena in the objective function. Empirical results presented here suggest that these phenomena are not significant factors in SGD optimization of MLP-related objective functions, and that the behavior of stochastic gradient descent in these problems is better described as the simultaneous convergence at different rates of many, largely non-interacting subproblems