Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studies
It provides a comprehensive review for researchers and practitioners working on distributed training of large neural networks, but is incremental as it synthesizes existing literature without new methods or results.
This survey addresses three research questions about model parallelism: categorizing types (intra-operator and inter-operator), identifying challenges (implementation and optimal partitioning), and examining modern use-cases in large transformer networks, based on limited public information.
Neural networks have become a cornerstone of machine learning. As the trend for these to get more and more complex continues, so does the underlying hardware and software infrastructure for training and deployment. In this survey we answer three research questions: "What types of model parallelism exist?", "What are the challenges of model parallelism?", and "What is a modern use-case of model parallelism?" We answer the first question by looking at how neural networks can be parallelised and expressing these as operator graphs while exploring the available dimensions. The dimensions along which neural networks can be parallelised are intra-operator and inter-operator. We answer the second question by collecting and listing both implementation challenges for the types of parallelism, as well as the problem of optimally partitioning the operator graph. We answer the last question by collecting and listing how parallelism is applied in modern multi-billion parameter transformer networks, to the extend that this is possible with the limited information shared about these networks.