Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information
This paper addresses the challenge of matching hidden neurons in federated learning for neural networks, which is a problem for researchers and practitioners working with distributed machine learning.
The authors propose a new federated learning method that extends Probabilistic Federated Neural Matching (PFNM) by incorporating a Kullback-Leibler divergence over neural components product. This approach allows for inference using posterior information from both local and global levels, and simulations show it outperforms state-of-the-art federated learning methods in both single and multiple communication rounds.
In federated learning, models trained on local clients are distilled into a global model. Due to the permutation invariance arises in neural networks, it is necessary to match the hidden neurons first when executing federated learning with neural networks. Through the Bayesian nonparametric framework, Probabilistic Federated Neural Matching (PFNM) matches and fuses local neural networks so as to adapt to varying global model size and the heterogeneity of the data. In this paper, we propose a new method which extends the PFNM with a Kullback-Leibler (KL) divergence over neural components product, in order to make inference exploiting posterior information in both local and global levels. We also show theoretically that The additional part can be seamlessly concatenated into the match-and-fuse progress. Through a series of simulations, it indicates that our new method outperforms popular state-of-the-art federated learning methods in both single communication round and additional communication rounds situation.