On the Variance of Temporal Difference Learning and its Reduction Using Control Variates
For reinforcement learning researchers, this provides theoretical understanding of TD variance and introduces DAE as a superior estimator.
The paper analyzes the variance of temporal difference (TD) learning, showing it is asymptotically bounded above by Monte Carlo estimators and that shorter horizons reduce variance. It also shows Direct Advantage Estimation (DAE) achieves a tighter variance bound than TD in the large-sample limit.
We analyze the variance of temporal difference (TD) learning using the phased setting with tabular representation, and show that one of the mechanisms behind its ability to reduce variance is by effectively aggregating over a larger number of independent trajectories. Based on this insight, we demonstrate that (1) the variance of TD is asymptotically bounded from above by Monte Carlo (MC) estimators, and (2) shorter horizon updates incurs less variance for a fixed number of samples. Beyond TD, we show that Direct Advantage Estimation (DAE), a method for estimating the advantage function, can be seen as a type of regression-adjusted control variate, which achieves a tighter bound on the variance compared to TD in the large-sample limit. Finally, we numerically illustrate the behaviors of these estimators with carefully designed environments.