CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging
This work addresses a key challenge in efficiently integrating multiple task-specific models into a multitask model, offering an incremental improvement over existing sparsification techniques.
The paper tackles the problem of conflicts between task vectors in model merging by proposing CABS, a framework that reduces parameter overlap and balances weight distribution through sparsification, resulting in outperforming state-of-the-art methods across diverse tasks and model sizes.
Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their performance: high parameter overlap and unbalanced weight distribution. To address these issues, we propose a simple, yet effective framework called CABS (Conflict-Aware and Balanced Sparsification), consisting of Conflict-Aware Sparsification (CA) and Balanced Sparsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$: $m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across diverse tasks and model sizes.