QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning
This work addresses computational efficiency in real-time planning for reinforcement learning, particularly in continuous control, though it is incremental as it builds on existing Transformer-based methods.
The paper tackles the computational burden of using Transformer Dynamics Models for long-horizon planning in reinforcement learning by proposing QT-TDM, which combines short-term planning with a Transformer Dynamics Model and long-term return estimation via an autoregressive Q-Transformer, resulting in superior performance and sample efficiency in continuous control tasks.
Inspired by the success of the Transformer architecture in natural language processing and computer vision, we investigate the use of Transformers in Reinforcement Learning (RL), specifically in modeling the environment's dynamics using Transformer Dynamics Models (TDMs). We evaluate the capabilities of TDMs for continuous control in real-time planning scenarios with Model Predictive Control (MPC). While Transformers excel in long-horizon prediction, their tokenization mechanism and autoregressive nature lead to costly planning over long horizons, especially as the environment's dimensionality increases. To alleviate this issue, we use a TDM for short-term planning, and learn an autoregressive discrete Q-function using a separate Q-Transformer (QT) model to estimate a long-term return beyond the short-horizon planning. Our proposed method, QT-TDM, integrates the robust predictive capabilities of Transformers as dynamics models with the efficacy of a model-free Q-Transformer to mitigate the computational burden associated with real-time planning. Experiments in diverse state-based continuous control tasks show that QT-TDM is superior in performance and sample efficiency compared to existing Transformer-based RL models while achieving fast and computationally efficient inference.