The Wisdom of Hindsight Makes Language Models Better Instruction Followers
This addresses the problem of simplifying instruction alignment for language models, offering a more efficient alternative to reinforcement learning methods, though it is incremental as it builds on existing supervised approaches.
The paper tackles the complexity of aligning language models with instructions by proposing Hindsight Instruction Relabeling (HIR), a reward-free method that converts feedback into relabeled instructions for supervised training, achieving performance comparable to or better than supervised fine-tuning on 12 BigBench reasoning tasks.
Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models. However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised manner. Such an algorithm doesn't require any additional parameters except for the original language model and maximally reuses the pretraining pipeline. To achieve this, we formulate instruction alignment problem for language models as a goal-reaching problem in decision making. We propose Hindsight Instruction Relabeling (HIR), a novel algorithm for aligning language models with instructions. The resulting two-stage algorithm shed light to a family of reward-free approaches that utilize the hindsightly relabeled instructions based on feedback. We evaluate the performance of HIR extensively on 12 challenging BigBench reasoning tasks and show that HIR outperforms the baseline algorithms and is comparable to or even surpasses supervised finetuning.