Mind the Gap: Data Rewriting for Stable Off-Policy Supervised Fine-Tuning
This work addresses a key challenge in off-policy learning for language model fine-tuning, offering a practical solution that improves stability and performance, though it is incremental as it builds on existing methods like importance sampling and KL penalties.
The paper tackles the problem of unstable optimization in supervised fine-tuning of large language models due to distribution mismatch between expert demonstrations and target policies, proposing a data rewriting framework that proactively shrinks this gap and achieves consistent and significant gains over existing methods on five mathematical reasoning benchmarks.
Supervised fine-tuning (SFT) of large language models can be viewed as an off-policy learning problem, where expert demonstrations come from a fixed behavior policy while training aims to optimize a target policy. Importance sampling is the standard tool for correcting this distribution mismatch, but large policy gaps lead to skewed weights, high variance, and unstable optimization. Existing methods mitigate this issue with KL penalties or clipping, which passively restrict updates rather than actively reducing the gap. We propose a simple yet effective data rewriting framework that proactively shrinks the policy gap before training. For each problem, correct model-generated solutions are kept as on-policy data, while incorrect ones are rewritten through guided re-solving, falling back to expert demonstrations only when needed. This aligns the training distribution with the target policy, reducing variance and improving stability. To handle residual mismatch after rewriting, we additionally apply importance sampling during training, forming a two-stage approach that combines data-level alignment with lightweight optimization-level correction. Experiments on five mathematical reasoning benchmarks show consistent and significant gains over both vanilla SFT and the state-of-the-art Dynamic Fine-Tuning (DFT) approach. Data and code will be released at https://github.com/NKU-HLT/Off-Policy-SFT.