Accelerated Policy Learning with Parallel Differentiable Simulation
This work addresses the problem of slow training in reinforcement learning for robotics and simulation-based control, offering a scalable solution for researchers and practitioners, though it is incremental by building on differentiable simulation methods.
The paper tackled the inefficiency of deep reinforcement learning in complex control tasks by developing a high-performance differentiable simulator and a new policy learning algorithm (SHAC) that improves sample efficiency and reduces training time, achieving over 17x faster training for muscle-actuated locomotion compared to established RL algorithms.
Deep reinforcement learning can generate complex control policies, but requires large amounts of training data to work effectively. Recent work has attempted to address this issue by leveraging differentiable simulators. However, inherent problems such as local minima and exploding/vanishing numerical gradients prevent these methods from being generally applied to control tasks with complex contact-rich dynamics, such as humanoid locomotion in classical RL benchmarks. In this work we present a high-performance differentiable simulator and a new policy learning algorithm (SHAC) that can effectively leverage simulation gradients, even in the presence of non-smoothness. Our learning algorithm alleviates problems with local minima through a smooth critic function, avoids vanishing/exploding gradients through a truncated learning window, and allows many physical environments to be run in parallel. We evaluate our method on classical RL control tasks, and show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms. In addition, we demonstrate the scalability of our method by applying it to the challenging high-dimensional problem of muscle-actuated locomotion with a large action space, achieving a greater than 17x reduction in training time over the best-performing established RL algorithm.