Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale Stochastic Approximation
This work addresses convergence issues in stochastic approximation methods, which are incremental for applications like reinforcement learning.
The paper tackles the finite-time complexity of linear two-time-scale stochastic approximation under time-varying step sizes and Markovian noise, showing mean square errors converge at a sublinear rate of O(k^{2/3}). It improves performance via a restarting scheme that matches constant step size complexity while ensuring exact convergence and preventing overly small step sizes.
Motivated by their broad applications in reinforcement learning, we study the linear two-time-scale stochastic approximation, an iterative method using two different step sizes for finding the solutions of a system of two equations. Our main focus is to characterize the finite-time complexity of this method under time-varying step sizes and Markovian noise. In particular, we show that the mean square errors of the variables generated by the method converge to zero at a sublinear rate $\Ocal(k^{2/3})$, where $k$ is the number of iterations. We then improve the performance of this method by considering the restarting scheme, where we restart the algorithm after every predetermined number of iterations. We show that using this restarting method the complexity of the algorithm under time-varying step sizes is as good as the one using constant step sizes, but still achieving an exact converge to the desired solution. Moreover, the restarting scheme also helps to prevent the step sizes from getting too small, which is useful for the practical implementation of the linear two-time-scale stochastic approximation.