Dynamic Model Merging Made Slim
For practitioners needing efficient multi-task model deployment, DiDi-Merging offers a more compact dynamic merging approach that improves accuracy-efficiency trade-offs.
DiDi-Merging achieves dynamic model merging with 1.24x parameters of a single fine-tuned model while matching prior dynamic baselines, and surpasses them at 1.4x, compared to methods requiring >2x storage. It applies across vision, language, and multimodal tasks.
Model merging enables the reuse of fine-tuned models without joint training or access to original data. Dynamic merging further improves flexibility by selectively activating task-relevant parameters and efficiently composing experts across multiple tasks. However, existing dynamic methods either maintain a full shared model with tiny experts or allocate excessive capacity to experts, leading to suboptimal accuracy--efficiency trade-offs. To address this, we propose DiDi-Merging, a slim dynamic merging framework that leverages differentiable rank allocation to balance shared and expert parameters. By formulating parameter budgeting as differentiable rank optimization in low-rank modules and introducing a data-free refinement step to recover task fidelity, DiDi-Merging matches prior dynamic baselines at only 1.24x the parameters of a single fine-tuned model and surpasses them at 1.4x, substantially more compact than methods requiring > 2x storage. DiDi-Merging applies across vision, language, and multimodal tasks.