Robustness and Regularization in Hierarchical Re-Basin
This work addresses model merging for AI practitioners, but it is incremental as it builds on existing Re-Basin techniques.
The paper investigates Git Re-Basin for merging trained models, proposing a hierarchical scheme that outperforms standard methods and induces robustness, but finds a larger performance drop than previously reported.
This paper takes a closer look at Git Re-Basin, an interesting new approach to merge trained models. We propose a hierarchical model merging scheme that significantly outperforms the standard MergeMany algorithm. With our new algorithm, we find that Re-Basin induces adversarial and perturbation robustness into the merged models, with the effect becoming stronger the more models participate in the hierarchical merging scheme. However, in our experiments Re-Basin induces a much bigger performance drop than reported by the original authors.