Power-law escape rate of SGD
This work addresses a theoretical problem in machine learning optimization, offering incremental insights into SGD behavior for researchers.
The paper tackled the problem of understanding SGD's escape rate from local minima by deriving a stochastic differential equation with additive noise, showing that the log loss barrier, not the linear loss barrier, determines escape rates, with rates depending on Hessian outlier eigenvalues. This result explains SGD's preference for flat minima with low effective dimensions, providing insight into implicit biases.
Stochastic gradient descent (SGD) undergoes complicated multiplicative noise for the mean-square loss. We use this property of SGD noise to derive a stochastic differential equation (SDE) with simpler additive noise by performing a random time change. Using this formalism, we show that the log loss barrier $Δ\log L=\log[L(θ^s)/L(θ^*)]$ between a local minimum $θ^*$ and a saddle $θ^s$ determines the escape rate of SGD from the local minimum, contrary to the previous results borrowing from physics that the linear loss barrier $ΔL=L(θ^s)-L(θ^*)$ decides the escape rate. Our escape-rate formula strongly depends on the typical magnitude $h^*$ and the number $n$ of the outlier eigenvalues of the Hessian. This result explains an empirical fact that SGD prefers flat minima with low effective dimensions, giving an insight into implicit biases of SGD.