Can Offline Reinforcement Learning Help Natural Language Understanding?
This work explores a novel connection between RL and language modeling for researchers in AI and NLP, though it appears incremental as it builds on existing pre-training methods.
The paper investigates whether offline reinforcement learning (RL) can aid natural language understanding by pre-training RL tasks with Transformers and evaluating on language tasks, finding that RL pre-trained models achieve performance close to those using language modeling objectives, indicating shared useful features across modalities.
Pre-training has been a useful method for learning implicit transferable knowledge and it shows the benefit of offering complementary features across different modalities. Recent work mainly focuses on the modalities such as image and text, for example, studies show that visual features learned from images can help visual-grounded language understanding. In this paper, we consider investigating the potential connection between offline reinforcement learning (RL) and language modeling (LM). Intuitively, RL and LM are similar in predicting the next states based on the current and previous states, which rely on both local and long-range dependency across states. To validate such an assumption, we pre-trained different offline RL tasks using Transformer and then evaluate these models on various language-related tasks. Experimental results show that our RL pre-trained models can give close performance compared with the models using the LM training objective, showing that there exist common useful features across these two modalities. To further explore the potential relationship, we investigate some factors such as Markov property and the sequential nature of RL trajectory.