GAPSL: A Gradient-Aligned Parallel Split Learning on Heterogeneous Data
This addresses the challenge of democratizing federated learning on resource-constrained devices by reducing training divergence, though it is incremental as it builds on existing parallel split learning methods.
The paper tackled the problem of training divergence in parallel split learning due to gradient directional inconsistency across clients, proposing GAPSL with leader gradient identification and gradient direction alignment to improve convergence, resulting in consistent outperformance in training accuracy and latency over state-of-the-art benchmarks.
The increasing complexity of neural networks poses significant challenges for democratizing FL on resource?constrained client devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. Since PSL is aggregation-free, it suffers from severe training divergence stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework that comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent client gradients to construct a leader gradient that captures the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy and latency.