Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approach
This addresses the problem of overfitting and limited personalization in federated learning for clients with small, heterogeneous data, though it appears incremental as it builds on existing personalized FL methods with a new theoretical approach.
The paper tackles the challenge of achieving high personalization in federated learning for heterogeneous clients with small datasets by introducing the PAC-PFL framework, which infers a shared hyper-posterior and personalizes client models without regularizing towards a single shared model, resulting in accurate and well-calibrated predictions as shown in experiments.
Federated Learning (FL) aims to infer a shared model from private and decentralized data stored by multiple clients. Personalized FL (PFL) enhances the model's fit for each client by adapting the global model to the clients. A significant level of personalization is required for highly heterogeneous clients but can be challenging to achieve, especially when clients' datasets are small. To address this issue, we introduce the PAC-PFL framework for PFL of probabilistic models. PAC-PFL infers a shared hyper-posterior and treats each client's posterior inference as the personalization step. Unlike previous PFL algorithms, PAC-PFL does not regularize all personalized models towards a single shared model, thereby greatly enhancing its personalization flexibility. By establishing and minimizing a PAC-Bayesian generalization bound on the average true loss of clients, PAC-PFL effectively mitigates overfitting even in data-poor scenarios. Additionally, PAC-PFL provides generalization bounds for new clients joining later. PAC-PFL achieves accurate and well-calibrated predictions, as supported by our experiments.