Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
This work addresses the challenge of improving fine-tuning methods for large language models and control policies, offering a more efficient alternative to advanced RL algorithms.
The paper tackles the problem of supervised fine-tuning (SFT) on curated data by showing it can be viewed as a form of reinforcement learning (RL) and proposes an importance-weighted variant (iw-SFT) that improves performance, achieving 66.7% on the AIME 2024 dataset.
Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.