Taylor TD-learning
This addresses variance reduction in TD-learning for reinforcement learning practitioners, but it is incremental as it builds on existing TD methods.
The paper tackles high variance in temporal-difference (TD) learning for reinforcement learning by introducing Taylor TD, a model-based framework that uses a first-order Taylor series expansion to reduce variance in continuous settings, showing it performs as well or better than state-of-the-art baselines on benchmark tasks.
Many reinforcement learning approaches rely on temporal-difference (TD) learning to learn a critic. However, TD-learning updates can be high variance. Here, we introduce a model-based RL framework, Taylor TD, which reduces this variance in continuous state-action settings. Taylor TD uses a first-order Taylor series expansion of TD updates. This expansion allows Taylor TD to analytically integrate over stochasticity in the action-choice, and some stochasticity in the state distribution for the initial state and action of each TD update. We include theoretical and empirical evidence that Taylor TD updates are indeed lower variance than standard TD updates. Additionally, we show Taylor TD has the same stable learning guarantees as standard TD-learning with linear function approximation under a reasonable assumption. Next, we combine Taylor TD with the TD3 algorithm, forming TaTD3. We show TaTD3 performs as well, if not better, than several state-of-the art model-free and model-based baseline algorithms on a set of standard benchmark tasks.