FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
This addresses communication inefficiencies for distributed training of LLMs, offering a significant performance improvement.
The paper tackles communication bottlenecks in distributed training of large language models by introducing FlashCommunication V2, a novel paradigm that enables efficient cross-GPU transmission at arbitrary bit widths, achieving up to 3.2x speedup in AllReduce and 2x in All2All communication.
Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2$\times$ speedup in AllReduce and 2$\times$ in All2All communication.