Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
This work addresses the need for simpler and more efficient DRL algorithms for researchers and practitioners in robotics and control, though it is incremental as it builds on existing methods like Soft Actor-Critic.
The paper tackled the problem of simplifying off-policy deep reinforcement learning (DRL) algorithms while maintaining high performance, by proposing output normalization and non-uniform sampling methods that matched or exceeded state-of-the-art sample efficiency on continuous control benchmarks.
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic. For standard continuous control benchmarks, Soft Actor-Critic (SAC), which employs entropy maximization, currently provides state-of-the-art performance. We first demonstrate that the entropy term in SAC addresses action saturation due to the bounded nature of the action spaces, with this insight, we propose a streamlined algorithm with a simple normalization scheme or with inverted gradients. We show that both approaches can match SAC's sample efficiency performance without the need of entropy maximization, we then propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. Extensive experimental results demonstrate that our proposed sampling scheme leads to state of the art sample efficiency on challenging continuous control tasks. We combine all of our findings into one simple algorithm, which we call Streamlined Off Policy with Emphasizing Recent Experience, for which we provide robust public-domain code.