Model Merging in the Essential Subspace
This addresses the problem of task interference for practitioners merging fine-tuned models, though it appears incremental as it builds on existing model merging research.
The paper tackles task interference in model merging by proposing ESM (Essential Subspace Merging), which uses PCA on feature shifts and multi-level polarized scaling to preserve task-specific functionality. The method achieves state-of-the-art performance in multi-task model merging across multiple task sets and model scales.
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging.