Domain-Adaptive Model Merging across Disconnected Modes
This addresses privacy and heterogeneity issues in distributed learning, though it is incremental as it builds on existing model merging techniques.
The paper tackles the problem of learning across domains without centralizing data by proposing DMM, a data-free model merging framework that consolidates knowledge from specialized models, achieving state-of-the-art performance on benchmarks.
Learning across domains is challenging when data cannot be centralized due to privacy or heterogeneity, which limits the ability to train a single comprehensive model. Model merging provides an appealing alternative by consolidating knowledge from multiple specialized models into one, avoiding data sharing and reducing retraining cost. In this work, we present DMM, a data-free model merging framework designed to handle highly divergent models. DMM proceeds in three steps. First, domain-specific models are trained independently. Second, models with high similarity are merged using standard techniques to ensure stability. Third, we synthesize pseudo-data from normalization statistics and distill knowledge from divergent models into the merged model through a lightweight refinement guided by these samples. This approach preserves rare but critical knowledge while maintaining stability. Extensive experiments on unimodal and multimodal benchmarks show that DMM achieves state-of-the-art performance over existing merging methods.