Deep Progressive Training: scaling up depth capacity of zero/one-layer models
This addresses the efficiency challenge in scaling deep learning models for practitioners, though it is incremental as it builds on progressive training methods.
The paper tackles the problem of high computational cost in training deep models by proposing zero/one-layer progressive training, which scales up model depth during training to save approximately 80% compute or accelerate 5x while achieving nearly the same loss as a fully trained 60-layer GPT2 model with 7B parameters.
Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost. To efficiently train models at scale, an effective strategy is the progressive training, which scales up model capacity during training, hence significantly reducing computation with little to none performance degradation. In this work, we study the depth expansion of large models through the lens of optimization theory and feature learning, offering insights on the initialization of new layers, hyperparameter transfer, learning rate schedule, and timing of model expansion. Specifically, we propose zero/one-layer progressive training for the optimal tradeoff between computation and loss. For example, zero/one-layer progressive training on GPT2 can save $\approx 80\%$ compute, or equivalently accelerate $\approx 5\times$ while achieving almost the same loss, compared to to a fully trained 60-layer model with 7B parameters.