Scaling Studies for Efficient Parameter Search and Parallelism for Large Language Model Pre-training
This work addresses computational and memory efficiency problems for researchers and engineers scaling LLMs, but it is incremental as it builds on existing parallelism techniques.
The study tackled the challenge of efficiently scaling large language model pre-training by analyzing three ML parallelism methods, specifically Microsoft DeepSpeed ZeRO stages, across models ranging from 580 million to 13 billion parameters, quantifying their relationships to optimize data processing and training.
AI accelerator processing capabilities and memory constraints largely dictate the scale in which machine learning workloads (e.g., training and inference) can be executed within a desirable time frame. Training a state of the art, transformer-based model today requires use of GPU-accelerated high performance computers with high-speed interconnects. As datasets and models continue to increase in size, computational requirements and memory demands for AI also continue to grow. These challenges have inspired the development of distributed algorithm and circuit-based optimization techniques that enable the ability to progressively scale models in multi-node environments, efficiently minimize neural network cost functions for faster convergence, and store more parameters into a set number of available resources. In our research project, we focus on parallel and distributed machine learning algorithm development, specifically for optimizing the data processing and pre-training of a set of 5 encoder-decoder LLMs, ranging from 580 million parameters to 13 billion parameters. We performed a fine-grained study to quantify the relationships between three ML parallelism methods, specifically exploring Microsoft DeepSpeed Zero Redundancy Optimizer (ZeRO) stages.