Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity
This addresses data heterogeneity issues in federated learning for distributed clients, offering a novel balance between personalization and global cooperation.
The paper tackles the problem of data heterogeneity in federated learning by introducing a partially personalized formulation that splits parameters into shared global and private local components, enabling perfect client data fitting and breaking the curse of heterogeneity in settings like local steps and Byzantine-robust training.
We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized. For instance, the shared global parameters can be used to learn good data representations, whereas the personalized layers are fine-tuned for a specific client. Moreover, we present a simple algorithm for the partially personalized formulation that offers significant benefits to all clients. In particular, it breaks the curse of data heterogeneity in several settings, such as training with local steps, asynchronous training, and Byzantine-robust training.