Tensor Decomposition based Personalized Federated Learning
This work addresses scalability and efficiency issues in federated learning for distributed machine learning applications, representing an incremental improvement with novel method elements.
The paper tackles the challenges of federated learning with statistical diversity and large-scale models by proposing TDPFed, a personalized FL framework using tensor decomposition to reduce communication costs, achieving state-of-the-art performance as demonstrated in experiments.
Federated learning (FL) is a new distributed machine learning framework that can achieve reliably collaborative training without collecting users' private data. However, due to FL's frequent communication and average aggregation strategy, they experience challenges scaling to statistical diversity data and large-scale models. In this paper, we propose a personalized FL framework, named Tensor Decomposition based Personalized Federated learning (TDPFed), in which we design a novel tensorized local model with tensorized linear layers and convolutional layers to reduce the communication cost. TDPFed uses a bi-level loss function to decouple personalized model optimization from the global model learning by controlling the gap between the personalized model and the tensorized local model. Moreover, an effective distributed learning strategy and two different model aggregation strategies are well designed for the proposed TDPFed framework. Theoretical convergence analysis and thorough experiments demonstrate that our proposed TDPFed framework achieves state-of-the-art performance while reducing the communication cost.