Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
This work addresses the problem of unfair evaluation baselines for researchers in reinforcement learning, highlighting an incremental but critical issue in benchmarking sample efficiency improvements.
The paper demonstrates that recent methods claiming improved data efficiency in deep reinforcement learning for Atari achieved gains primarily by allowing more training updates per data sample, not from novel techniques, and shows that a modified DQN baseline can match or exceed these results with lower complexity and computational costs.
Reinforcement learning (RL) has seen great advancements in the past few years. Nevertheless, the consensus among the RL community is that currently used methods, despite all their benefits, suffer from extreme data inefficiency, especially in the rich visual domains like Atari. To circumvent this problem, novel approaches were introduced that often claim to be much more efficient than popular variations of the state-of-the-art DQN algorithm. In this paper, however, we demonstrate that the newly proposed techniques simply used unfair baselines in their experiments. Namely, we show that the actual improvement in the efficiency came from allowing the algorithm for more training updates for each data sample, and not from employing the new methods. By allowing DQN to execute network updates more frequently we manage to reach similar or better results than the recently proposed advancement, often at a fraction of complexity and computational costs. Furthermore, based on the outcomes of the study, we argue that the agent similar to the modified DQN that is presented in this paper should be used as a baseline for any future work aimed at improving sample efficiency of deep reinforcement learning.