Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
This addresses communication efficiency for federated learning practitioners, but it is incremental as it builds on existing model merging and interpolation ideas.
The paper tackles the high communication costs in adapting large pre-trained models in federated learning by proposing Local Superior Soups, a model interpolation-based local training technique that enhances performance and reduces communication rounds across diverse datasets.
Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.'' Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets. Our code is available at \href{https://github.com/ubc-tea/Local-Superior-Soups}{https://github.com/ubc-tea/Local-Superior-Soups}.