Online Decision Transformer
This addresses a practical challenge in reinforcement learning for researchers and practitioners by enabling more efficient online adaptation from offline data, though it is incremental as it builds on existing sequence modeling approaches.
The paper tackles the problem of integrating offline pretraining with online finetuning in reinforcement learning, proposing Online Decision Transformers (ODT) which achieves competitive state-of-the-art performance on the D4RL benchmark and shows significant gains during finetuning.
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framework uses sequence-level entropy regularizers in conjunction with autoregressive modeling objectives for sample-efficient exploration and finetuning. Empirically, we show that ODT is competitive with the state-of-the-art in absolute performance on the D4RL benchmark but shows much more significant gains during the finetuning procedure.