Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
This work addresses a bottleneck in network-intensive high-performance computing systems like public clouds, offering a practical improvement for distributed training efficiency.
The paper tackles the problem of convergence deterioration in multi-hop all-reduce distributed training when using one-bit compressed stochastic gradient descent, and proposes Marsit, a sign-bit compression framework that reduces training time by up to 35% while maintaining accuracy.
Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression.