Never Worse, Mostly Better: Stable Policy Improvement in Deep Reinforcement Learning
This addresses convergence issues for researchers and practitioners in RL, offering a stable improvement method, though it is incremental as it builds on existing conservative update approaches.
The paper tackled the problem of instability and abrupt performance deterioration in deep reinforcement learning by proposing EVEREST, a method that provides conservative updates using confidence bounds of a reference policy, resulting in significant improvements in continuous control and Atari benchmarks.
In recent years, there has been significant progress in applying deep reinforcement learning (RL) for solving challenging problems across a wide variety of domains. Nevertheless, convergence of various methods has been shown to suffer from inconsistencies, due to algorithmic instability and variance, as well as stochasticity in the benchmark environments. Particularly, despite the fact that the agent's performance may be improving on average, it may abruptly deteriorate at late stages of training. In this work, we study methods for enhancing the agent's learning process, by providing conservative updates with respect to either the obtained history or a reference benchmark policy. Our method, termed EVEREST, obtains high confidence improvements via confidence bounds of a reference policy. Through extensive empirical analysis we demonstrate the benefit of our approach in terms of both performance and stabilization, with significant improvements in continuous control and Atari benchmarks.