Efficient Construction of Model Family through Progressive Training Using Model Expansion
This addresses the problem of high computational costs for AI practitioners needing model families, offering an incremental improvement over standard independent training methods.
The paper tackles the computational inefficiency of independently training each model in a family of different sizes by proposing a progressive training method that expands smaller models to larger ones, reducing costs by about 25% while maintaining comparable performance and achieving more consistent behavior across sizes.
As Large Language Models (LLMs) gain widespread practical application, providing the model family of different parameter sizes has become standard practice to address diverse computational requirements. Conventionally, each model in a family is trained independently, resulting in computational costs that scale additively with the number of models. We propose an efficient method for constructing the model family through progressive training, where smaller models are incrementally expanded to larger sizes to create a complete model family. Through extensive experiments with a model family ranging from 1B to 8B parameters, we demonstrate that our method reduces computational costs by approximately 25% while maintaining comparable performance to independently trained models. Furthermore, by strategically adjusting maximum learning rates based on model size, our method outperforms the independent training across various metrics. Beyond performance gains, our approach offers an additional advantage: models in our family tend to yield more consistent behavior across different model sizes.