The Impact of Cut Layer Selection in Split Federated Learning
This provides a quantitative analysis for practitioners in distributed machine learning, though it is incremental as it builds on existing SFL frameworks.
The paper tackles the problem of how cut layer selection affects model performance in Split Federated Learning (SFL), finding that SFL-V1 is invariant to cut layer choice while SFL-V2's performance varies significantly, with optimal selection outperforming FedAvg on heterogeneous data.
Split Federated Learning (SFL) is a distributed machine learning paradigm that combines federated learning and split learning. In SFL, a neural network is partitioned at a cut layer, with the initial layers deployed on clients and remaining layers on a training server. There are two main variants of SFL: SFL-V1 where the training server maintains separate server-side models for each client, and SFL-V2 where the training server maintains a single shared model for all clients. While existing studies have focused on algorithm development for SFL, a comprehensive quantitative analysis of how the cut layer selection affects model performance remains unexplored. This paper addresses this gap by providing numerical and theoretical analysis of SFL performance and convergence relative to cut layer selection. We find that SFL-V1 is relatively invariant to the choice of cut layer, which is consistent with our theoretical results. Numerical experiments on four datasets and two neural networks show that the cut layer selection significantly affects the performance of SFL-V2. Moreover, SFL-V2 with an appropriate cut layer selection outperforms FedAvg on heterogeneous data.