On the Effects of Fine-tuning Language Models for Text-Based Reinforcement Learning
This addresses the problem of optimizing language model fine-tuning for text-based RL agents, which is incremental as it builds on prior work by highlighting specific risks and benefits.
The paper investigates the role of semantic understanding in text-based reinforcement learning, showing that it leads to efficient training but can cause semantic degeneration from inappropriate fine-tuning, affecting performance on similar tasks.
Text-based reinforcement learning involves an agent interacting with a fictional environment using observed text and admissible actions in natural language to complete a task. Previous works have shown that agents can succeed in text-based interactive environments even in the complete absence of semantic understanding or other linguistic capabilities. The success of these agents in playing such games suggests that semantic understanding may not be important for the task. This raises an important question about the benefits of LMs in guiding the agents through the game states. In this work, we show that rich semantic understanding leads to efficient training of text-based RL agents. Moreover, we describe the occurrence of semantic degeneration as a consequence of inappropriate fine-tuning of language models in text-based reinforcement learning (TBRL). Specifically, we describe the shift in the semantic representation of words in the LM, as well as how it affects the performance of the agent in tasks that are semantically similar to the training games. We believe these results may help develop better strategies to fine-tune agents in text-based RL scenarios.