Realistic Evaluation of Model Merging for Compositional Generalization
This work addresses the need for standardized evaluation in model merging for researchers and practitioners, but it is incremental as it focuses on benchmarking rather than introducing new methods.
The paper tackled the problem of evaluating model merging methods for compositional generalization by comparing them in a shared experimental setting across image classification, image generation, and NLP, and found that it clarified the state of the field and provided a rigorous setup for testing new methods.
Merging has become a widespread way to cheaply combine individual models into a single model that inherits their capabilities and attains better performance. This popularity has spurred rapid development of many new merging methods, which are typically validated in disparate experimental settings and frequently differ in the assumptions made about model architecture, data availability, and computational budget. In this work, we characterize the relative merits of different merging methods by evaluating them in a shared experimental setting and precisely identifying the practical requirements of each method. Specifically, our setting focuses on using merging for compositional generalization of capabilities in image classification, image generation, and natural language processing. Additionally, we measure the computational costs of different merging methods as well as how they perform when scaling the number of models being merged. Taken together, our results clarify the state of the field of model merging and provide a comprehensive and rigorous experimental setup to test new methods.