On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
This work addresses the impact of pre-training modalities on offline RL performance, offering insights for researchers in reinforcement learning and transfer learning, though it is incremental in nature.
The study investigated how pre-training Transformers on language or vision data affects fine-tuning for offline reinforcement learning tasks, finding that language pre-training enables efficient learning even without context, while image pre-training suffers from gradient issues and less adaptation.
We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks. Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one. A closer look at the parameter changes of the pre-trained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping. To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information. Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.