Closing the gap between SVRG and TD-SVRG with Gradient Splitting
This work addresses a specific bottleneck in reinforcement learning policy evaluation by closing the performance gap between TD learning and SVRG, representing an incremental improvement over existing methods.
The paper tackles the problem of improving the convergence rate of Temporal Difference (TD) learning in reinforcement learning by integrating it with the Stochastic Variance Reduced Gradient (SVRG) method, achieving a geometric convergence bound with a learning rate of 1/8, matching SVRG's performance in convex optimization.
Temporal difference (TD) learning is a policy evaluation in reinforcement learning whose performance can be enhanced by variance reduction methods. Recently, multiple works have sought to fuse TD learning with Stochastic Variance Reduced Gradient (SVRG) method to achieve a geometric rate of convergence. However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization. In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG. Our main result is a geometric convergence bound with predetermined learning rate of $1/8$, which is identical to the convergence bound available for SVRG in the convex setting. Our theoretical findings are supported by a set of experiments.