Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learning
This work addresses scalability issues in split learning for decentralized learning with private data, offering a hybrid approach that combines benefits of SL and FL, though it appears incremental as it builds on existing hybrid methods like SplitFed.
The authors tackled the scalability bottlenecks in split learning (SL) by proposing SGLR, a framework that uses server-side local gradient averaging and split learning rates, achieving higher accuracy than baseline SL methods, including SplitFed, and matching federated learning (FL) performance with lower energy and communication costs.
In recent years, there have been great advances in the field of decentralized learning with private data. Federated learning (FL) and split learning (SL) are two spearheads possessing their pros and cons, and are suited for many user clients and large models, respectively. To enjoy both benefits, hybrid approaches such as SplitFed have emerged of late, yet their fundamentals have still been illusive. In this work, we first identify the fundamental bottlenecks of SL, and thereby propose a scalable SL framework, coined SGLR. The server under SGLR broadcasts a common gradient averaged at the split-layer, emulating FL without any additional communication across clients as opposed to SplitFed. Meanwhile, SGLR splits the learning rate into its server-side and client-side rates, and separately adjusts them to support many clients in parallel. Simulation results corroborate that SGLR achieves higher accuracy than other baseline SL methods including SplitFed, which is even on par with FL consuming higher energy and communication costs. As a secondary result, we observe greater reduction in leakage of sensitive information via mutual information using SLGR over the baselines.