Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training
This work addresses the problem of poor OOD generalization in VLMs for researchers and practitioners, offering an incremental improvement by introducing a data-centric method to bridge the performance gap between SFT and RL.
The paper tackles the generalization gap in Vision-Language Models (VLMs) where Reinforcement Learning (RL) outperforms Supervised Fine-Tuning (SFT) on out-of-distribution (OOD) tasks, attributing it to RL's implicit filtering of medium-difficulty samples and proposing Difficulty-Curated SFT (DC-SFT) to explicitly filter training data, which enhances OOD generalization over standard SFT and surpasses RL-based training with greater stability and efficiency.
The adaptation of large-scale Vision-Language Models (VLMs) through post-training reveals a pronounced generalization gap: models fine-tuned with Reinforcement Learning (RL) consistently achieve superior out-of-distribution (OOD) performance compared to those trained with Supervised Fine-Tuning (SFT). This paper posits a data-centric explanation for this phenomenon, contending that RL's generalization advantage arises from an implicit data filtering mechanism that inherently prioritizes medium-difficulty training samples. To test this hypothesis, we systematically evaluate the OOD generalization of SFT models across training datasets of varying difficulty levels. Our results confirm that data difficulty is a critical factor, revealing that training on hard samples significantly degrades OOD performance. Motivated by this finding, we introduce Difficulty-Curated SFT (DC-SFT), a straightforward method that explicitly filters the training set based on sample difficulty. Experiments show that DC-SFT not only substantially enhances OOD generalization over standard SFT, but also surpasses the performance of RL-based training, all while providing greater stability and computational efficiency. This work offers a data-centric account of the OOD generalization gap in VLMs and establishes a more efficient pathway to achieving robust generalization. Code is available at https://github.com/byyx666/DC-SFT.