Adapting Double Q-Learning for Continuous Reinforcement Learning
This addresses a fundamental issue in reinforcement learning for continuous control tasks, though it is incremental as it builds on Double Q-Learning.
The paper tackles overestimation bias in off-policy reinforcement learning by proposing a novel bias correction method using a mixture policy with two components, each evaluated by separate networks, achieving near-state-of-the-art results on a small set of MuJoCo environments.
Majority of off-policy reinforcement learning algorithms use overestimation bias control techniques. Most of these techniques rooted in heuristics, primarily addressing the consequences of overestimation rather than its fundamental origins. In this work we present a novel approach to the bias correction, similar in spirit to Double Q-Learning. We propose using a policy in form of a mixture with two components. Each policy component is maximized and assessed by separate networks, which removes any basis for the overestimation bias. Our approach shows promising near-SOTA results on a small set of MuJoCo environments.