AdaRank: Adaptive Rank Pruning for Enhanced Model Merging
This work addresses cross-task interference in model merging for multi-task learning, offering an incremental improvement over existing SVD-based techniques.
The paper tackles the problem of suboptimal performance in model merging due to manually designed rank selection, proposing AdaRank which adaptively prunes singular components to reduce cross-task interference, achieving state-of-the-art performance and reducing the performance gap to nearly 1%.
Model merging has emerged as a promising approach for unifying independently fine-tuned models into an integrated framework, significantly enhancing computational efficiency in multi-task learning. Recently, several SVD-based techniques have been introduced to exploit low-rank structures for enhanced merging, but their reliance on such manually designed rank selection often leads to cross-task interference and suboptimal performance. In this paper, we propose AdaRank, a novel model merging framework that adaptively selects the most beneficial singular directions of task vectors to merge multiple models. We empirically show that the dominant singular components of task vectors can cause critical interference with other tasks, and that naive truncation across tasks and layers degrades performance. In contrast, AdaRank dynamically prunes the singular components that cause interference and offers an optimal amount of information to each task vector by learning to prune ranks during test-time via entropy minimization. Our analysis demonstrates that such method mitigates detrimental overlaps among tasks, while empirical results show that AdaRank consistently achieves state-of-the-art performance with various backbones and number of tasks, reducing the performance gap between fine-tuned models to nearly 1%.