Systems for Parallel and Distributed Large-Model Deep Learning Training
It provides a comprehensive overview for researchers and practitioners dealing with scale-driven issues in deep learning training, but it is incremental as it surveys existing methods.
This survey addresses the systems challenges of training large deep learning models, such as memory bottlenecks and high costs, by exploring parallelization and memory-efficient techniques.
Deep learning (DL) has transformed applications in a variety of domains, including computer vision, natural language processing, and tabular data analysis. The search for improved DL model accuracy has led practitioners to explore increasingly large neural architectures, with some recent Transformer models spanning hundreds of billions of learnable parameters. These designs have introduced new scale-driven systems challenges for the DL space, such as memory bottlenecks, poor runtime efficiency, and high costs of model development. Efforts to address these issues have explored techniques such as parallelization of neural architectures, spilling data across the memory hierarchy, and memory-efficient data representations. This survey will explore the large-model training systems landscape, highlighting key challenges and the various techniques that have been used to address them.