Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism
This addresses a critical problem for researchers and practitioners scaling large models in decentralized settings, representing a novel method rather than an incremental improvement.
The paper tackles the challenge of communication bottlenecks in decentralized model-parallel training by proposing a novel compression algorithm that compresses activations and gradients, achieving up to 99% compression with no convergence degradation and enabling up to 100x improvement in communication efficiency, allowing training of billion-parameter models over low-end internet connections.
Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.