Transient learning dynamics drive escape from sharp valleys in Stochastic Gradient Descent
This provides a foundational physical framework linking dynamics, geometry, and generalization in deep learning, potentially guiding the design of better optimization algorithms.
The study tackled the problem of understanding why Stochastic Gradient Descent (SGD) prefers flatter solutions in deep learning by analyzing its learning dynamics, revealing that SGD noise reshapes the loss landscape to favor flat minima and a transient freezing mechanism traps dynamics in these regions, with increased noise delaying freezing to enhance convergence.
Stochastic gradient descent (SGD) is central to deep learning, yet the dynamical origin of its preference for flatter, more generalizable solutions remains unclear. Here, by analyzing SGD learning dynamics, we identify a nonequilibrium mechanism governing solution selection. Numerical experiments reveal a transient exploratory phase in which SGD trajectories repeatedly escape sharp valleys and transition toward flatter regions of the loss landscape. By using a tractable physical model, we show that the SGD noise reshapes the landscape into an effective potential that favors flat solutions. Crucially, we uncover a transient freezing mechanism: as training proceeds, growing energy barriers suppress inter-valley transitions and ultimately trap the dynamics within a single basin. Increasing the SGD noise strength delays this freezing, which enhances convergence to flatter minima. Together, these results provide a unified physical framework linking learning dynamics, loss-landscape geometry, and generalization, and suggest principles for the design of more effective optimization algorithms.