Mixing of Stochastic Accelerated Gradient Descent
This work addresses optimization efficiency for machine learning practitioners, but it is incremental as it builds on existing SGD and acceleration methods.
The paper tackles the mixing properties of stochastic accelerated gradient descent (SAGD) for least-squares regression, showing that SAGD mixes faster than SGD with an explicit rate depending on data moments, supported by theoretical proofs and experiments.
We study the mixing properties for stochastic accelerated gradient descent (SAGD) on least-squares regression. First, we show that stochastic gradient descent (SGD) and SAGD are simulating the same invariant distribution. Motivated by this, we then establish mixing rate for SAGD-iterates and compare it with those of SGD-iterates. Theoretically, we prove that the chain of SAGD iterates is geometrically ergodic --using a proper choice of parameters and under regularity assumptions on the input distribution. More specifically, we derive an explicit mixing rate depending on the first 4 moments of the data distribution. By means of illustrative examples, we prove that SAGD-iterate chain mixes faster than the chain of iterates obtained by SGD. Furthermore, we highlight applications of the established mixing rate in the convergence analysis of SAGD on realizable objectives. The proposed analysis is based on a novel non-asymptotic analysis of products of random matrices. This theoretical result is substantiated and validated by experiments.