How neural networks find generalizable solutions: Self-tuned annealing in deep learning
This work addresses a fundamental gap in theoretical understanding of SGD for researchers in machine learning, though it appears incremental as it builds on existing SGD analysis.
The paper tackled the problem of understanding how Stochastic Gradient Descent (SGD) finds generalizable solutions in deep learning by analyzing learning dynamics and loss landscapes, discovering an inverse relation between weight variance and landscape flatness, and developing a random landscape theory to explain it, with results indicating SGD uses a self-tuned annealing strategy to locate flat minima.
Despite the tremendous success of Stochastic Gradient Descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions in the high-dimensional weight space. By analyzing the learning dynamics and loss function landscape, we discover a robust inverse relation between the weight variance and the landscape flatness (inverse of curvature) for all SGD-based learning algorithms. To explain the inverse variance-flatness relation, we develop a random landscape theory, which shows that the SGD noise strength (effective temperature) depends inversely on the landscape flatness. Our study indicates that SGD attains a self-tuned landscape-dependent annealing strategy to find generalizable solutions at the flat minima of the landscape. Finally, we demonstrate how these new theoretical insights lead to more efficient algorithms, e.g., for avoiding catastrophic forgetting.