When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
This provides explainable guidance for fine-tuning LLMs, addressing a practical problem for NLP practitioners by optimizing performance across context lengths, though it is incremental in exploring SFT effects.
The study tackled how supervised fine-tuning (SFT) data length affects large language models (LLMs) on short-context tasks, finding that long-context SFT improves short-context performance, contrary to degradation from long-context pretraining, and that hybrid training mitigates knowledge bias.
Large language models (LLMs) have achieved impressive performance across natural language processing (NLP) tasks. As real-world applications increasingly demand longer context windows, continued pretraining and supervised fine-tuning (SFT) on long-context data has become a common approach. While the effects of data length in continued pretraining have been extensively studied, their implications for SFT remain unclear. In this work, we systematically investigate how SFT data length influences LLM behavior on short-context tasks. Counterintuitively, we find that long-context SFT improves short-context performance, contrary to the commonly observed degradation from long-context pretraining. To uncover the underlying mechanisms of this phenomenon, we first decouple and analyze two key components, Multi-Head Attention (MHA) and Feed-Forward Network (FFN), and show that both independently benefit from long-context SFT. We further study their interaction and reveal a knowledge preference bias: long-context SFT promotes contextual knowledge, while short-context SFT favors parametric knowledge, making exclusive reliance on long-context SFT suboptimal. Finally, we demonstrate that hybrid training mitigates this bias, offering explainable guidance for fine-tuning LLMs.