On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
This work addresses the problem of fine-tuning language models without catastrophic forgetting for researchers and practitioners in NLP, offering incremental improvements by integrating concepts from RM into DM.
The paper explores theoretical connections between Reward Maximization (RM) and Distribution Matching (DM) for fine-tuning language models, showing that methods like KL-control can be viewed as part of DM and addressing training difficulties such as high gradient variance by importing the concept of a baseline into DM. It empirically validates this approach on controllable language generation tasks, observing superior performance in constraint satisfaction, stability, and sample efficiency.
The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a training-from-scratch to a fine-tuning paradigm. While in some applications the goal is to "nudge" the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the theoretical connections between the two paradigms, and show that methods such as KL-control developed for RM can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability and sample efficiency.