CALM: Consensus-Aware Localized Merging for Multi-Task Learning
This addresses the challenge of model merging for multi-task learning, offering a more effective method for practitioners, though it appears incremental as it builds on existing global- and local-aware approaches.
The paper tackles the problem of merging multiple fine-tuned models in multi-task learning by proposing CALM, which integrates localized information with global task consensus to avoid parameter interference and preserve task-specific details, achieving performance close to traditional multi-task learning.
Model merging aims to integrate the strengths of multiple fine-tuned models into a unified model while preserving task-specific capabilities. Existing methods, represented by task arithmetic, are typically classified into global- and local-aware methods. However, global-aware methods inevitably cause parameter interference, while local-aware methods struggle to maintain the effectiveness of task-specific details in the merged model. To address these limitations, we propose a Consensus-Aware Localized Merging (CALM) method which incorporates localized information aligned with global task consensus, ensuring its effectiveness post-merging. CALM consists of three key components: (1) class-balanced entropy minimization sampling, providing a more flexible and reliable way to leverage unsupervised data; (2) an efficient-aware framework, selecting a small set of tasks for sequential merging with high scalability; (3) a consensus-aware mask optimization, aligning localized binary masks with global task consensus and merging them conflict-free. Experiments demonstrate the superiority and robustness of our CALM, significantly outperforming existing methods and achieving performance close to traditional MTL.