Mixture of Step Returns in Bootstrapped DQN
This addresses a methodological limitation in DRL for improving learning efficiency and exploration, but appears incremental as it builds on existing bootstrapped DQN frameworks.
The paper tackles the problem of sacrificing diversity when integrating different step returns into a single target in deep reinforcement learning, proposing Mixture Bootstrapped DQN (MB-DQN) which uses different backup lengths for different bootstrapped heads. The result is improved performance over baseline methods on Atari 2600 benchmarks, though no concrete numbers are provided in the abstract.
The concept of utilizing multi-step returns for updating value functions has been adopted in deep reinforcement learning (DRL) for a number of years. Updating value functions with different backup lengths provides advantages in different aspects, including bias and variance of value estimates, convergence speed, and exploration behavior of the agent. Conventional methods such as TD-lambda leverage these advantages by using a target value equivalent to an exponential average of different step returns. Nevertheless, integrating step returns into a single target sacrifices the diversity of the advantages offered by different step return targets. To address this issue, we propose Mixture Bootstrapped DQN (MB-DQN) built on top of bootstrapped DQN, and uses different backup lengths for different bootstrapped heads. MB-DQN enables heterogeneity of the target values that is unavailable in approaches relying only on a single target value. As a result, it is able to maintain the advantages offered by different backup lengths. In this paper, we first discuss the motivational insights through a simple maze environment. In order to validate the effectiveness of MB-DQN, we perform experiments on the Atari 2600 benchmark environments, and demonstrate the performance improvement of MB-DQN over a number of baseline methods. We further provide a set of ablation studies to examine the impacts of different design configurations of MB-DQN.