Beyond Parameter Arithmetic: Sparse Complementary Fusion for Distribution-Aware Model Merging
This addresses the issue of unstable generation and poor generalization in model merging for AI practitioners, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackles the problem of interference and degraded performance in model merging for large language models by proposing Sparse Complementary Fusion with reverse KL (SCF-RKL), which uses distribution-aware updates to selectively integrate complementary parameters, resulting in consistent outperformance over existing methods across 24 benchmarks.
Model merging has emerged as a promising paradigm for composing the capabilities of large language models by directly operating in weight space, enabling the integration of specialized models without costly retraining. However, existing merging methods largely rely on parameter-space heuristics, which often introduce severe interference, leading to degraded generalization and unstable generation behaviors such as repetition and incoherent outputs. In this work, we propose Sparse Complementary Fusion with reverse KL (SCF-RKL), a novel model merging framework that explicitly controls functional interference through sparse, distribution-aware updates. Instead of assuming linear additivity in parameter space, SCF-RKL measures the functional divergence between models using reverse Kullback-Leibler divergence and selectively incorporates complementary parameters. This mode-seeking, sparsity-inducing design effectively preserves stable representations while integrating new capabilities. We evaluate SCF-RKL across a wide range of model scales and architectures, covering both reasoning-focused and instruction-tuned models. Extensive experiments on 24 benchmarks spanning advanced reasoning, general reasoning and knowledge, instruction following, and safety demonstrate, vision classification that SCF-RKL consistently outperforms existing model merging methods while maintaining strong generalization and generation stability.