C3-SL: Circular Convolution-Based Batch-Wise Compression for Communication-Efficient Split Learning
This work addresses communication bottlenecks in split learning for distributed machine learning systems, presenting a novel compression approach that is incremental but offers substantial efficiency gains.
The paper tackles the problem of communication inefficiency in split learning by proposing a batch-wise compression method that compresses multiple features into one using circular convolution, achieving a 16x compression ratio with negligible accuracy drops on CIFAR datasets and significantly reducing memory and computation overhead compared to existing methods.
Most existing studies improve the efficiency of Split learning (SL) by compressing the transmitted features. However, most works focus on dimension-wise compression that transforms high-dimensional features into a low-dimensional space. In this paper, we propose circular convolution-based batch-wise compression for SL (C3-SL) to compress multiple features into one single feature. To avoid information loss while merging multiple features, we exploit the quasi-orthogonality of features in high-dimensional space with circular convolution and superposition. To the best of our knowledge, we are the first to explore the potential of batch-wise compression under the SL scenario. Based on the simulation results on CIFAR-10 and CIFAR-100, our method achieves a 16x compression ratio with negligible accuracy drops compared with the vanilla SL. Moreover, C3-SL significantly reduces 1152x memory and 2.25x computation overhead compared to the state-of-the-art dimension-wise compression method.