RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning
This addresses the problem of costly or wasteful data usage in LLM alignment for researchers and practitioners, offering a more data-efficient approach.
The paper tackles the data inefficiency in LLM alignment by proposing RIFT, a framework that repurposes negative samples via reward-informed fine-tuning, achieving consistent performance improvements over existing methods like RFT on mathematical benchmarks.
While Supervised Fine-Tuning (SFT) and Rejection Sampling Fine-Tuning (RFT) are standard for LLM alignment, they either rely on costly expert data or discard valuable negative samples, leading to data inefficiency. To address this, we propose Reward Informed Fine-Tuning (RIFT), a simple yet effective framework that utilizes all self-generated samples. Unlike the hard thresholding of RFT, RIFT repurposes negative trajectories, reweighting the loss with scalar rewards to learn from both the positive and negative trajectories from the model outputs. To overcome the training collapse caused by naive reward integration, where direct multiplication yields an unbounded loss, we introduce a stabilized loss formulation that ensures numerical robustness and optimization efficiency. Extensive experiments on mathematical benchmarks across various base models show that RIFT consistently outperforms RFT. Our results demonstrate that RIFT is a robust and data-efficient alternative for alignment using mixed-quality, self-generated data.