Power-law Dynamic arising from machine learning
This work addresses theoretical understanding of optimization dynamics in machine learning, but it appears incremental as it focuses on analyzing a specific SDE without broad application or empirical validation.
The paper tackles the analysis of a new stochastic differential equation (SDE) derived from machine learning optimization, called the power-law dynamic, which has a stationary distribution with power-law tails. It proves ergodicity and a unique stationary distribution under small learning rates, and compares exit times between continuous and discretized versions to guide algorithm design.
We study a kind of new SDE that was arisen from the research on optimization in machine learning, we call it power-law dynamic because its stationary distribution cannot have sub-Gaussian tail and obeys power-law. We prove that the power-law dynamic is ergodic with unique stationary distribution, provided the learning rate is small enough. We investigate its first exist time. In particular, we compare the exit times of the (continuous) power-law dynamic and its discretization. The comparison can help guide machine learning algorithm.