Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
This work addresses the problem of high computational cost in deep reinforcement learning research, which creates barriers to entry for researchers with limited resources, by advocating for the continued use of small-scale environments.
This paper argues that small-scale environments can still provide valuable scientific insights in deep reinforcement learning, despite the community's focus on large-scale benchmarks. To support this, the authors re-examine the Rainbow algorithm and present new insights into its constituent algorithms.
Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.