How does the optimizer implicitly bias the model merging loss landscape?
This work addresses the problem of understanding and improving model merging for machine learning practitioners, providing insights into how training dynamics influence merging success, though it is incremental in building on prior connections between optimizer noise and loss landscape properties.
The paper investigates how optimization choices affect the geometry of the loss landscape and their impact on the success of model merging methods, showing that a single quantity called effective noise scale unifies these effects and exhibits a non-monotonic relationship with merging effectiveness across architectures and datasets.
Model merging methods combine models with different capabilities into a single one while maintaining the same inference cost. Two popular approaches are linear interpolation, which linearly interpolates between model weights, and task arithmetic, which combines task vectors obtained by the difference between finetuned and base models. While useful in practice, what properties make merging effective are poorly understood. This paper explores how the optimization process affects the loss landscape geometry and its impact on merging success. We show that a single quantity -- the effective noise scale -- unifies the impact of optimizer and data choices on model merging. Across architectures and datasets, the effectiveness of merging success is a non-monotonic function of effective noise, with a distinct optimum. Decomposing this quantity, we find that larger learning rates, stronger weight decay, smaller batch sizes, and data augmentation all independently modulate the effective noise scale, exhibiting the same qualitative trend. Unlike prior work that connects optimizer noise to the flatness or generalization of individual minima, we show that it also affects the global loss landscape, predicting when independently trained solutions can be merged. Our findings broaden the understanding of how optimization shapes the loss landscape geometry and its downstream consequences for model merging, suggesting the possibility of further manipulating the training dynamics to improve merging effectiveness.