Efficient Continuous Control with Double Actors and Regularized Critics
This addresses bias issues in value estimation for continuous control RL, offering incremental improvements over existing methods like DDPG and TD3.
The paper tackles the problem of over- and underestimation bias in value estimation for reinforcement learning by introducing double actors and regularized critics, resulting in the DARC algorithm that significantly outperforms state-of-the-art methods with higher sample efficiency on continuous control tasks.
How to obtain good value estimation is one of the key problems in Reinforcement Learning (RL). Current value estimation methods, such as DDPG and TD3, suffer from unnecessary over- or underestimation bias. In this paper, we explore the potential of double actors, which has been neglected for a long time, for better value function estimation in continuous setting. First, we uncover and demonstrate the bias alleviation property of double actors by building double actors upon single critic and double critics to handle overestimation bias in DDPG and underestimation bias in TD3 respectively. Next, we interestingly find that double actors help improve the exploration ability of the agent. Finally, to mitigate the uncertainty of value estimate from double critics, we further propose to regularize the critic networks under double actors architecture, which gives rise to Double Actors Regularized Critics (DARC) algorithm. Extensive experimental results on challenging continuous control tasks show that DARC significantly outperforms state-of-the-art methods with higher sample efficiency.