NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learning
This work addresses communication efficiency in distributed machine learning for scenarios with limited bandwidth, representing an incremental improvement over existing split learning methods.
The paper tackles the communication overhead in split learning by proposing NSC-SL, a bandwidth-aware adaptive compression algorithm that dynamically adjusts low-rank approximation based on singular values and uses error-compensated tensor factorization, achieving high compression ratios while preserving information for convergence.
The expanding scale of neural networks poses a major challenge for distributed machine learning, particularly under limited communication resources. While split learning (SL) alleviates client computational burden by distributing model layers between clients and server, it incurs substantial communication overhead from frequent transmission of intermediate activations and gradients. To tackle this issue, we propose NSC-SL, a bandwidth-aware adaptive compression algorithm for communication-efficient SL. NSC-SL first dynamically determines the optimal rank of low-rank approximation based on the singular value distribution for adapting real-time bandwidth constraints. Then, NSC-SL performs error-compensated tensor factorization using alternating orthogonal iteration with residual feedback, effectively minimizing truncation loss. The collaborative mechanisms enable NSC-SL to achieve high compression ratios while preserving semantic-rich information essential for convergence. Extensive experiments demonstrate the superb performance of NSC-SL.