Personalised Federated Learning: A Combinational Approach
This work addresses performance variability and degradation in federated learning for clients with non-i.i.d. data and privacy constraints, representing an incremental improvement through combination of existing methods.
The paper tackles performance degradation in federated learning due to non-i.i.d. data and privacy mechanisms like differential privacy and robust aggregation, finding that combining personalization approaches such as finetuning and knowledge distillation improves performance for most clients, recovering from these issues.
Federated learning (FL) is a distributed machine learning approach involving multiple clients collaboratively training a shared model. Such a system has the advantage of more training data from multiple clients, but data can be non-identically and independently distributed (non-i.i.d.). Privacy and integrity preserving features such as differential privacy (DP) and robust aggregation (RA) are commonly used in FL. In this work, we show that on common deep learning tasks, the performance of FL models differs amongst clients and situations, and FL models can sometimes perform worse than local models due to non-i.i.d. data. Secondly, we show that incorporating DP and RA degrades performance further. Then, we conduct an ablation study on the performance impact of different combinations of common personalization approaches for FL, such as finetuning, mixture-of-experts ensemble, multi-task learning, and knowledge distillation. It is observed that certain combinations of personalization approaches are more impactful in certain scenarios while others always improve performance, and combination approaches are better than individual ones. Most clients obtained better performance with combined personalized FL and recover from performance degradation caused by non-i.i.d. data, DP, and RA.