A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
This provides practical guidance for applying transformers in online RL, but it is incremental as it focuses on design optimizations rather than a breakthrough.
The paper tackled the challenge of using transformers in online model-free reinforcement learning for continuous control, showing they can achieve competitive performance across various tasks.
Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.