Distributed Hybrid Parallelism for Large Language Models: Comparative Study and System Design Guide
It addresses the challenge of efficiently distributing computation and memory for large language models, offering insights for researchers and practitioners, but it is incremental as it builds on existing surveys with added analysis and case studies.
This paper provides a comprehensive review and comparative study of distributed parallel strategies for large language models, analyzing their benefits and trade-offs to guide the design of optimal systems for efficient training and inference.
With the rapid growth of large language models (LLMs), a wide range of methods have been developed to distribute computation and memory across hardware devices for efficient training and inference. While existing surveys provide descriptive overviews of these techniques, systematic analysis of their benefits and trade offs and how such insights can inform principled methodology for designing optimal distributed systems remain limited. This paper offers a comprehensive review of collective operations and distributed parallel strategies, complemented by mathematical formulations to deepen theoretical understanding. We further examine hybrid parallelization designs, emphasizing communication computation overlap across different stages of model deployment, including both training and inference. Recent advances in automated search for optimal hybrid parallelization strategies using cost models are also discussed. Moreover, we present case studies with mainstream architecture categories to reveal empirical insights to guide researchers and practitioners in parallelism strategy selection. Finally, we highlight open challenges and limitations of current LLM training paradigms and outline promising directions for the next generation of large scale model development.