MIN-Merging: Merge the Important Neurons for Model Merging
This addresses the parameter conflict problem in model merging for deep learning practitioners, offering a practical solution with incremental improvements.
The paper tackled the problem of parameter conflicts degrading performance in model merging by proposing MIN-Merging, a router-based framework that selectively merges important neurons, resulting in consistent gains on in-domain tasks while retaining generalization on out-of-domain tasks.
Recent advances in deep learning have led to a surge of open-source models across diverse domains. While model merging offers a promising way to combine their strengths, existing approaches often suffer from parameter conflicts that degrade performance on domain-specific tasks. We propose MIN-Merging, a router-based framework that selectively merges the most important neurons to reduce such conflicts. Extensive experiments on Computer Vision(CV) and Natural Language Processing(NLP) benchmarks show that MIN-Merging achieves consistent gains on in-domain tasks while retaining the generalization ability of pretrained models on out-of-domain tasks. These results highlight its effectiveness as a practical solution to the parameter conflict problem in model merging.