Extrapolation Merging: Keep Improving With Extrapolation and Merging
This addresses the challenge of efficient post-fine-tuning enhancement for LLMs, though it is incremental as it builds on existing model merging techniques.
The paper tackles the problem of improving large language models after instruction fine-tuning without additional computational resources or data by proposing Extrapolation Merging, which uses model extrapolation to guide merging and achieves consistent performance gains across seven tasks.
Large Language Models (LLMs) require instruction fine-tuning to perform different downstream tasks. However, the instruction fine-tuning phase still demands significant computational resources and labeled data, lacking a paradigm that can improve model performance without additional computational power and data. Model merging aims to enhance performance by combining the parameters of different models, but the lack of a clear optimization direction during the merging process does not always guarantee improved performance. In this paper, we attempt to provide a clear optimization direction for model merging. We first validate the effectiveness of the model extrapolation method during the instruction fine-tuning phase. Then, we propose Extrapolation Merging, a paradigm that can continue improving model performance without requiring extra computational resources or data. Using the extrapolation method, we provide a clear direction for model merging, achieving local optimization search, and consequently enhancing the merged model's performance. We conduct experiments on seven different tasks, and the results show that our method can consistently improve the model's performance after fine-tuning.