PLayer-FL: A Principled Approach to Personalized Layer-wise Cross-Silo Federated Learning
This work addresses data heterogeneity in cross-silo Federated Learning, offering a personalized approach that is incremental by building on partial FL methods.
The paper tackled the challenge of non-identically distributed data in Federated Learning by introducing PLayer-FL, a method that uses a federation sensitivity metric to identify layers benefiting from federation, which outperformed existing FL algorithms on various tasks and achieved more uniform performance improvements across clients.
Non-identically distributed data is a major challenge in Federated Learning (FL). Personalized FL tackles this by balancing local model adaptation with global model consistency. One variant, partial FL, leverages the observation that early layers learn more transferable features by federating only early layers. However, current partial FL approaches use predetermined, architecture-specific rules to select layers, limiting their applicability. We introduce Principled Layer-wise-FL (PLayer-FL), which uses a novel federation sensitivity metric to identify layers that benefit from federation. This metric, inspired by model pruning, quantifies each layer's contribution to cross-client generalization after the first training epoch, identifying a transition point in the network where the benefits of federation diminish. We first demonstrate that our federation sensitivity metric shows strong correlation with established generalization measures across diverse architectures. Next, we show that PLayer-FL outperforms existing FL algorithms on a range of tasks, also achieving more uniform performance improvements across clients.