Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods
This addresses a specific bias issue in deep reinforcement learning for continuous control, offering incremental improvements over existing methods.
The paper tackles the underestimation bias in deep actor-critic reinforcement learning methods caused by high-variance reinforcement signals, introducing a parameter-free Q-learning variant that combines Clipped Double Q-learning and Maxmin Q-learning to bound value estimates, resulting in state-of-the-art improvements across OpenAI Gym continuous control tasks.
In value-based deep reinforcement learning methods, approximation of value functions induces overestimation bias and leads to suboptimal policies. We show that in deep actor-critic methods that aim to overcome the overestimation bias, if the reinforcement signals received by the agent have a high variance, a significant underestimation bias arises. To minimize the underestimation, we introduce a parameter-free, novel deep Q-learning variant. Our Q-value update rule combines the notions behind Clipped Double Q-learning and Maxmin Q-learning by computing the critic objective through the nested combination of maximum and minimum operators to bound the approximate value estimates. We evaluate our modification on the suite of several OpenAI Gym continuous control tasks, improving the state-of-the-art in every environment tested.