Bootstrapped Transformer for Offline Reinforcement Learning
This work addresses a key bottleneck in offline RL for researchers and practitioners by enhancing data coverage, though it is incremental as it builds on existing sequence modeling approaches.
The paper tackles the problem of limited and poorly covered datasets in offline reinforcement learning by proposing Bootstrapped Transformer, which uses bootstrapping to generate additional pseudo data for training sequence models, resulting in improved performance that beats strong baselines on two benchmarks.
Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment. Recent works provide a novel perspective by viewing offline RL as a generic sequence generation problem, adopting sequence models such as Transformer architecture to model distributions over trajectories, and repurposing beam search as a planning algorithm. However, the training datasets utilized in general offline RL tasks are quite limited and often suffer from insufficient distribution coverage, which could be harmful to training sequence generation models yet has not drawn enough attention in the previous works. In this paper, we propose a novel algorithm named Bootstrapped Transformer, which incorporates the idea of bootstrapping and leverages the learned model to self-generate more offline data to further boost the sequence model training. We conduct extensive experiments on two offline RL benchmarks and demonstrate that our model can largely remedy the existing offline RL training limitations and beat other strong baseline methods. We also analyze the generated pseudo data and the revealed characteristics may shed some light on offline RL training. The codes are available at https://seqml.github.io/bootorl.