FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning
This addresses a key challenge for institutions with computational constraints in federated learning, though it appears incremental as it builds on existing device-heterogeneity approaches.
The paper tackles the problem of training a large global model using only smaller local models in device-heterogeneity federated learning, known as the small-to-large scenario, and proposes FedBRB, which achieves state-of-the-art results and can outperform baselines even with minimal local models.
Recently, the success of large models has demonstrated the importance of scaling up model size. This has spurred interest in exploring collaborative training of large-scale models from federated learning perspective. Due to computational constraints, many institutions struggle to train a large-scale model locally. Thus, training a larger global model using only smaller local models has become an important scenario (i.e., the \textbf{small-to-large scenario}). Although recent device-heterogeneity federated learning approaches have started to explore this area, they face limitations in fully covering the parameter space of the global model. In this paper, we propose a method called \textbf{FedBRB} (\underline{B}lock-wise \underline{R}olling and weighted \underline{B}roadcast) based on the block concept. FedBRB can uses small local models to train all blocks of the large global model, and broadcasts the trained parameters to the entire space for faster information interaction. Experiments demonstrate FedBRB yields substantial performance gains, achieving state-of-the-art results in this scenario. Moreover, FedBRB using only minimal local models can even surpass baselines using larger local models.