Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning
This addresses the challenge of data quality reliance in fine-tuning LLMs, offering a method to enhance performance without requiring high-quality data, though it appears incremental as it builds on existing SFT approaches.
The paper tackles the problem of supervised fine-tuning (SFT) for large language models, which often suffers from performance degradation due to poor data quality, by introducing a forgetting mechanism that categorizes tokens as positive or negative and explicitly forgets negative ones, resulting in improved overall performance and more diverse responses on established benchmarks.
Supervised fine-tuning (SFT) plays a critical role for pretrained large language models (LLMs), notably enhancing their capacity to acquire domain-specific knowledge while preserving or potentially augmenting their general-purpose capabilities. However, the efficacy of SFT hinges on data quality as well as data volume, otherwise it may result in limited performance gains or even degradation relative to the associated baselines. To mitigate such reliance, we suggest categorizing tokens within each corpus into two parts -- positive and negative tokens -- based on whether they are useful to improve model performance. Positive tokens can be trained in common ways, whereas negative tokens, which may lack essential semantics or be misleading, should be explicitly forgotten. Overall, the token categorization facilitate the model to learn less informative message, and the forgetting process shapes a knowledge boundary to guide the model on what information to learn more precisely. We conduct experiments on well-established benchmarks, finding that this forgetting mechanism not only improves overall model performance and also facilitate more diverse model responses.