FREE-Merging: Fourier Transform for Efficient Model Merging
This addresses the challenge of efficiently integrating diverse pre-trained models for users needing adaptable AI systems, though it appears incremental as it builds on existing model merging methods.
The paper tackles the problem of task interference in model merging by revealing its presence in the frequency domain and proposing FR-Merging to filter harmful interference with minimal computational overhead, achieving balanced trade-offs in training cost, inference latency, storage, and performance across CV, NLP, and Multi-Modal tasks.
With the rapid growth of deep learning, there is an increasing availability of open-source models for various tasks. However, single fine-tuned models often fall short of meeting the diverse needs of users. Model merging has thus emerged as an efficient method to integrate the capabilities of existing models into a unified model. Nevertheless, existing model merging methods face challenging trade-offs between performance and deployment costs, primarily due to task interference. For the first time, we reveal that task interference is evident in the frequency domain of model parameters, yet current efforts only focus on spatial domain solutions, which are largely ineffective in addressing frequency domain interference. To mitigate the impact of frequency domain interference, we propose FR-Merging, an innovative method that effectively filters harmful frequency domain interference on the backbone with minimal computational overhead. Since performance loss is inevitable with cost-free methods, we propose a lightweight task-specific expert module that dynamically compensates for information loss during merging. This proposed framework, FREE-Merging (FR-Merging with experts), strikes a balanced trade-off between training cost, inference latency, storage requirements, and performance. We demonstrate the effectiveness of both FR-Merging and FREE-Merging on multiple tasks across CV, NLP, and Multi-Modal domains and show that they can be flexibly adapted to specific needs.