Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
For researchers in reinforcement learning, this work provides clear guidance on which demonstration-augmentation strategies are most effective for sample-efficient online RL, resolving ambiguity from prior conflicting results.
This paper categorizes demonstration-augmented RL methods into three types and empirically evaluates their combinations, finding that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining for improving online sample efficiency.
Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.