Distributional Reinforcement Learning for Efficient Exploration
This work addresses the challenge of exploration in RL for applications like gaming and autonomous driving, presenting an incremental improvement over existing distributional RL methods.
The paper tackles the problem of efficient exploration in deep reinforcement learning by proposing a method that uses a decaying schedule to suppress intrinsic uncertainty and an exploration bonus from upper quantiles of the learned distribution. It results in outperforming QR-DQN in Atari 2600 games with a 483% average gain in cumulative rewards and achieving near-optimal safety rewards twice as fast in a CARLA driving simulator.
In distributional reinforcement learning (RL), the estimated distribution of value function models both the parametric and intrinsic uncertainties. We propose a novel and efficient exploration method for deep RL that has two components. The first is a decaying schedule to suppress the intrinsic uncertainty. The second is an exploration bonus calculated from the upper quantiles of the learned distribution. In Atari 2600 games, our method outperforms QR-DQN in 12 out of 14 hard games (achieving 483 \% average gain across 49 games in cumulative rewards over QR-DQN with a big win in Venture). We also compared our algorithm with QR-DQN in a challenging 3D driving simulator (CARLA). Results show that our algorithm achieves near-optimal safety rewards twice faster than QRDQN.