Model Merging with Functional Dual Anchors
This work addresses model merging for efficient post-training integration of multiple finetuned models, though it appears incremental as it builds on existing parameter-space methods.
The paper tackles the problem of parameter inconsistencies in model merging by proposing Functional Dual Anchors (FDAs), which model the input-representation space instead of the parameter space, resulting in improved robustness and flexibility for integrating knowledge from multiple finetuned checkpoints.
Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input-representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-space model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.