How to Combine Variational Bayesian Networks in Federated Learning
This addresses the need for uncertainty quantification in safety-critical federated learning applications, though it is incremental as it builds on existing probabilistic models.
The paper tackles the problem of aggregating variational Bayesian neural networks in federated learning, where traditional methods for deterministic models fail due to weight distributions, and finds that the spread of aggregated distributions significantly impacts learning, with empirical benchmarks on three image classification datasets.
Federated Learning enables multiple data centers to train a central model collaboratively without exposing any confidential data. Even though deterministic models are capable of performing high prediction accuracy, their lack of calibration and capability to quantify uncertainty is problematic for safety-critical applications. Different from deterministic models, probabilistic models such as Bayesian neural networks are relatively well-calibrated and able to quantify uncertainty alongside their competitive prediction accuracy. Both of the approaches appear in the federated learning framework; however, the aggregation scheme of deterministic models cannot be directly applied to probabilistic models since weights correspond to distributions instead of point estimates. In this work, we study the effects of various aggregation schemes for variational Bayesian neural networks. With empirical results on three image classification datasets, we observe that the degree of spread for an aggregated distribution is a significant factor in the learning process. Hence, we present an investigation on the question of how to combine variational Bayesian networks in federated learning, while providing benchmarks for different aggregation settings.