Bayesian Federated Model Compression for Communication and Computation Efficiency
This work addresses efficiency challenges in federated learning for distributed systems, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of communication and computation inefficiency in federated learning by proposing a Bayesian model compression framework, resulting in significant reductions in both communication overhead and computational complexity as shown in simulations.
In this paper, we investigate Bayesian model compression in federated learning (FL) to construct sparse models that can achieve both communication and computation efficiencies. We propose a decentralized Turbo variational Bayesian inference (D-Turbo-VBI) FL framework where we firstly propose a hierarchical sparse prior to promote a clustered sparse structure in the weight matrix. Then, by carefully integrating message passing and VBI with a decentralized turbo framework, we propose the D-Turbo-VBI algorithm which can (i) reduce both upstream and downstream communication overhead during federated training, and (ii) reduce the computational complexity during local inference. Additionally, we establish the convergence property for thr proposed D-Turbo-VBI algorithm. Simulation results show the significant gain of our proposed algorithm over the baselines in reducing communication overhead during federated training and computational complexity of final model.