Rethinking Data Selection for Supervised Fine-Tuning
This work addresses the challenge of improving SFT efficiency and effectiveness for AI alignment, though it is incremental by introducing a simple heuristic based on response length.
The paper tackles the problem of data selection for supervised fine-tuning (SFT) of large language models by proposing that focusing on human-like interactions, rather than data quality or diversity, is key. It finds that selecting instances with long responses leads to superior downstream performance compared to using full datasets or other selection methods.
Although supervised finetuning (SFT) has emerged as an essential technique to align large language models with humans, it is considered superficial, with style learning being its nature. At the same time, recent works indicate the importance of data selection for SFT, showing that finetuning with high-quality and diverse subsets of the original dataset leads to superior downstream performance. In this work, we rethink the intuition behind data selection for SFT. Considering SFT is superficial, we propose that essential demonstrations for SFT should focus on reflecting human-like interactions instead of data quality or diversity. However, it is not straightforward to directly assess to what extent a demonstration reflects human styles. Towards an initial attempt in this direction, we find selecting instances with long responses is surprisingly more effective for SFT than utilizing full datasets or instances selected based on quality and diversity. We hypothesize that such a simple heuristic implicitly mimics a crucial aspect of human-style conversation: detailed responses are usually more helpful.