Leveraging Submodule Linearity Enhances Task Arithmetic Performance in LLMs
This work addresses the challenge of effective multi-task model merging for AI practitioners, offering an incremental improvement over existing methods.
The paper tackled the problem of improving multi-task model merging via task arithmetic by showing that submodules within models exhibit higher linearity than the overall model, and proposed a merging strategy with closed-form weights that consistently outperforms standard approaches across different scales and tasks.
Task arithmetic is a straightforward yet highly effective strategy for model merging, enabling the resultant model to exhibit multi-task capabilities. Recent research indicates that models demonstrating linearity enhance the performance of task arithmetic. In contrast to existing methods that rely on the global linearization of the model, we argue that this linearity already exists within the model's submodules. In particular, we present a statistical analysis and show that submodules (e.g., layers, self-attentions, and MLPs) exhibit significantly higher linearity than the overall model. Based on these findings, we propose an innovative model merging strategy that independently merges these submodules. Especially, we derive a closed-form solution for optimal merging weights grounded in the linear properties of these submodules. Experimental results demonstrate that our method consistently outperforms the standard task arithmetic approach and other established baselines across different model scales and various tasks. This result highlights the benefits of leveraging the linearity of submodules and provides a new perspective for exploring solutions for effective and practical multi-task model merging.