Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States
This work addresses the challenge of balancing exploration and exploitation in reinforcement learning for continuous control, offering an incremental improvement over existing methods.
The paper tackles the problem of improving exploration in actor-critic algorithms by introducing a method that incentivizes plausible novel states, leading to enhanced sample efficiency and training performance in continuous control tasks.
Actor-critic (AC) algorithms are a class of model-free deep reinforcement learning algorithms, which have proven their efficacy in diverse domains, especially in solving continuous control problems. Improvement of exploration (action entropy) and exploitation (expected return) using more efficient samples is a critical issue in AC algorithms. A basic strategy of a learning algorithm is to facilitate indiscriminately exploring all of the environment state space, as well as to encourage exploring rarely visited states rather than frequently visited one. Under this strategy, we propose a new method to boost exploration through an intrinsic reward, based on measurement of a state's novelty and the associated benefit of exploring the state (with regards to policy optimization), altogether called plausible novelty. With incentivized exploration of plausible novel states, an AC algorithm is able to improve its sample efficiency and hence training performance. The new method is verified by extensive simulations of continuous control tasks of MuJoCo environments on a variety of prominent off-policy AC algorithms.