Ctrl-Z: Recovering from Instability in Reinforcement Learning
This addresses instability in reinforcement learning for real-world robotic control, offering incremental improvements in data efficiency and stability.
The paper tackled the problem of unstable training dynamics in reinforcement learning, particularly for safety-sensitive robotics, by proposing a model-agnostic approach that reverts to previous agent parameters when performance decreases, showing improvements over state-of-the-art algorithms like DDPG in 5 out of 6 environments and no decrease with TD3.
When learning behavior, training data is often generated by the learner itself; this can result in unstable training dynamics, and this problem has particularly important applications in safety-sensitive real-world control tasks such as robotics. In this work, we propose a principled and model-agnostic approach to mitigate the issue of unstable learning dynamics by maintaining a history of a reinforcement learning agent over the course of training, and reverting to the parameters of a previous agent whenever performance significantly decreases. We develop techniques for evaluating this performance through statistical hypothesis testing of continued improvement, and evaluate them on a standard suite of challenging benchmark tasks involving continuous control of simulated robots. We show improvements over state-of-the-art reinforcement learning algorithms in performance and robustness to hyperparameters, outperforming DDPG in 5 out of 6 evaluation environments and showing no decrease in performance with TD3, which is known to be relatively stable. In this way, our approach takes an important step towards increasing data efficiency and stability in training for real-world robotic applications.