The Prospect of Enhancing Large-Scale Heterogeneous Federated Learning with Transformers
This work addresses scalability and heterogeneity issues in federated learning for distributed data owners, but it is incremental as it applies an existing method (Transformers) to a new context.
The paper tackles the challenges of data heterogeneity and large-scale data owners in federated learning by investigating Transformer-based models, finding that Transformers outperform ResNet and personalized ResNet approaches in generalization and personalization across varying numbers of data owners.
Federated learning (FL) addresses data privacy concerns by enabling collaborative training of AI models across distributed data owners. Wide adoption of FL faces the fundamental challenges of data heterogeneity and the large scale of data owners involved. In this paper, we investigate the prospect of Transformer-based FL models for achieving generalization and personalization in this setting. We conduct extensive comparative experiments involving FL with Transformers, ResNet, and personalized ResNet-based FL approaches under various scenarios. These experiments consider varying numbers of data owners to demonstrate Transformers' advantages over deep neural networks in large-scale heterogeneous FL tasks. In addition, we analyze the superior performance of Transformers by comparing the Centered Kernel Alignment (CKA) representation similarity across different layers and FL models to gain insight into the reasons behind their promising capabilities.