Stochastic Variance Reduction for Deep Q-learning
This addresses sample efficiency and training stability issues for deep reinforcement learning practitioners, but it is incremental as it applies an existing optimization technique to a specific domain.
The paper tackled the problem of poor gradient estimation with excessive variance in deep Q-learning, which causes unstable training and poor sample efficiency, by proposing an optimization strategy using stochastic variance reduced gradient (SVRG) techniques, resulting in outperforming baselines on 18 out of 20 Atari games.
Recent advances in deep reinforcement learning have achieved human-level performance on a variety of real-world applications. However, the current algorithms still suffer from poor gradient estimation with excessive variance, resulting in unstable training and poor sample efficiency. In our paper, we proposed an innovative optimization strategy by utilizing stochastic variance reduced gradient (SVRG) techniques. With extensive experiments on Atari domain, our method outperforms the deep q-learning baselines on 18 out of 20 games.