Does compressing activations help model parallel training?
This addresses the challenge of slow training due to communication in model parallelism for researchers and practitioners, though it is incremental as it applies existing compression techniques to a new setting.
The paper tackles the problem of communication bottlenecks in model-parallel training of large-scale Transformer models by empirically evaluating the effectiveness of compression methods, finding that these methods can improve training speed across over 160 settings and 8 datasets.
Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensive model parallelism. One way to improve training speed is to compress the message size in communication. Previous approaches have primarily focused on compressing gradients in a data parallelism setting, but compression in a model-parallel setting is an understudied area. We have discovered that model parallelism has fundamentally different characteristics than data parallelism. In this work, we present the first empirical study on the effectiveness of compression methods for model parallelism. We implement and evaluate three common classes of compression algorithms - pruning-based, learning-based, and quantization-based - using a popular Transformer training framework. We evaluate these methods across more than 160 settings and 8 popular datasets, taking into account different hyperparameters, hardware, and both fine-tuning and pre-training stages. We also provide analysis when the model is scaled up. Finally, we provide insights for future development of model parallelism compression algorithms.