Toward Data Efficient Model Merging between Different Datasets without Performance Degradation
This addresses a data efficiency challenge for practitioners in machine learning who need to merge models from different datasets without performance degradation, representing an incremental advancement in model merging methods.
The paper tackles the problem of merging models trained on different datasets, which previously required full datasets and incurred high computational costs, by showing that dataset reduction techniques like coreset selection and dataset condensation can significantly improve accuracy, achieving a 31% improvement with full datasets and 24% with sampled subsets in SPLIT-CIFAR10 experiments.
Model merging is attracting attention as a novel method for creating a new model by combining the weights of different trained models. While previous studies reported that model merging works well for models trained on a single dataset with different random seeds, model merging between different datasets remains unsolved. In this paper, we attempt to reveal the difficulty in merging such models trained on different datasets and alleviate it. Our empirical analyses show that, in contrast to the single-dataset scenarios, dataset information needs to be accessed to achieve high accuracy when merging models trained on different datasets. However, the requirement to use full datasets not only incurs significant computational costs but also becomes a major limitation when integrating models developed and shared by others. To address this, we demonstrate that dataset reduction techniques, such as coreset selection and dataset condensation, effectively reduce the data requirement for model merging. In our experiments with SPLIT-CIFAR10 model merging, the accuracy is significantly improved by $31%$ when using the full dataset and $24%$ when using the sampled subset compared with not using the dataset.