Reducing Communication for Split Learning by Randomized Top-k Sparsification
This work addresses communication bottlenecks in split learning for VFL applications, offering an incremental improvement over existing compression techniques.
The paper tackles communication inefficiency in split learning for Vertical Federated Learning by proposing randomized top-k sparsification, which improves model generalization and convergence by probabilistically selecting elements, achieving better performance than other methods at the same compression level.
Split learning is a simple solution for Vertical Federated Learning (VFL), which has drawn substantial attention in both research and application due to its simplicity and efficiency. However, communication efficiency is still a crucial issue for split learning. In this paper, we investigate multiple communication reduction methods for split learning, including cut layer size reduction, top-k sparsification, quantization, and L1 regularization. Through analysis of the cut layer size reduction and top-k sparsification, we further propose randomized top-k sparsification, to make the model generalize and converge better. This is done by selecting top-k elements with a large probability while also having a small probability to select non-top-k elements. Empirical results show that compared with other communication-reduction methods, our proposed randomized top-k sparsification achieves a better model performance under the same compression level.