Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
This work provides a theoretical framework for LLM alignment, but it is incremental as it builds on existing SFT and DPO methods.
The paper tackles the problem of unifying Supervised Fine-Tuning (SFT) and preference learning in LLM post-training by showing they operate in the same optimal policy-reward subspace, and addresses a limitation in SFT by proposing a learning rate reduction method that yields up to 25% relative gain and 6% absolute win rate increase in instruction following tasks.
Post-training processes are essential phases in grounding pre-trained language models to real-world tasks, with learning from demonstrations or preference signals playing a crucial role in this adaptation. We present a unified theoretical framework bridging Supervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training. Through rigorous mathematical derivation, we demonstrate that both SFT and preference learning methods like Direct Preference Optimization (DPO) operate within the same optimal policy-reward subspace, with SFT representing a special case of implicit reward learning. Our analysis reveals a critical limitation in conventional SFT: the KL divergence term in distribution matching becomes constant with respect to the policy during optimization, failing to constrain model updates. To address this, we propose a simple yet effective learning rate reduction approach that yields significant performance improvements (up to \textbf{25\%} relative gain and \textbf{6\%} absolute win rate increase in instruction following tasks. Additionally, we derive alternative SFT objectives from various f-divergence functions that preserve the KL term during optimization, further enhancing post-DPO model performance. Finally, we extend the theoretical relationship between LLM logits and Q-functions from preference learning to the SFT context, providing mathematical derivations and experimental validation.