A Generative Framework for Personalized Learning and Estimation: Theory, Algorithms, and Privacy
This work addresses the challenge of data heterogeneity in federated learning for personalized model training, offering incremental improvements through a unifying framework and new algorithms.
The authors tackled the problem of statistical heterogeneity in federated learning by proposing a generative framework that unifies existing personalized learning algorithms and suggests new ones, resulting in a new algorithm AdaPeD that numerically outperforms known methods and includes privacy guarantees with user-level protection.
A distinguishing characteristic of federated learning is that the (local) client data could have statistical heterogeneity. This heterogeneity has motivated the design of personalized learning, where individual (personalized) models are trained, through collaboration. There have been various personalization methods proposed in literature, with seemingly very different forms and methods ranging from use of a single global model for local regularization and model interpolation, to use of multiple global models for personalized clustering, etc. In this work, we begin with a generative framework that could potentially unify several different algorithms as well as suggest new algorithms. We apply our generative framework to personalized estimation, and connect it to the classical empirical Bayes' methodology. We develop private personalized estimation under this framework. We then use our generative framework for learning, which unifies several known personalized FL algorithms and also suggests new ones; we propose and study a new algorithm AdaPeD based on a Knowledge Distillation, which numerically outperforms several known algorithms. We also develop privacy for personalized learning methods with guarantees for user-level privacy and composition. We numerically evaluate the performance as well as the privacy for both the estimation and learning problems, demonstrating the advantages of our proposed methods.