Demystifying the Communication Characteristics for Distributed Transformer Models
This addresses communication bottlenecks in distributed training of transformer models, which is crucial for scaling large models like LLMs, though it is incremental in nature.
The paper analyzes communication patterns in distributed transformer training, identifying small message point-to-point communication as a key bottleneck and revealing correlations between sequence length, throughput, model size, and optimizations.
Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication behavior of transformer models - that is, how different parallelism schemes used in multi-node/multi-GPU DL Training communicate data in the context of transformers. We use GPT-based language models as a case study of the transformer architecture due to their ubiquity. We validate the empirical results obtained from our communication logs using analytical models. At a high level, our analysis reveals a need to optimize small message point-to-point communication further, correlations between sequence length, per-GPU throughput, model size, and optimizations used, and where to potentially guide further optimizations in framework and HPC middleware design and optimization.