DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
This work simplifies training for researchers and practitioners in robotics and control by addressing instabilities and reducing sample complexity, though it is incremental as it builds on existing off-policy methods.
The paper tackles the complexity and sample inefficiency in continuous-control off-policy reinforcement learning by showing that simple deterministic policy gradient methods work well with bias control, identifying policy update instabilities, and using importance sampling to prevent performance deterioration, achieving competitive results on MuJoCo tasks.
This paper prescribes a suite of techniques for off-policy Reinforcement Learning (RL) that simplify the training process and reduce the sample complexity. First, we show that simple Deterministic Policy Gradient works remarkably well as long as the overestimation bias is controlled. This is contrast to existing literature which creates sophisticated off-policy techniques. Second, we pinpoint training instabilities, typical of off-policy algorithms, to the greedy policy update step; existing solutions such as delayed policy updates do not mitigate this issue. Third, we show that ideas in the propensity estimation literature can be used to importance-sample transitions from the replay buffer and selectively update the policy to prevent deterioration of performance. We make these claims using extensive experimentation on a set of challenging MuJoCo tasks. A short video of our results can be seen at https://tinyurl.com/scs6p5m .