Speeding up Deep Model Training by Sharing Weights and Then Unsharing
This addresses training efficiency for large language models like BERT, though it appears incremental as it builds on existing BERT architecture with a modified training procedure.
The paper tackles the problem of slow BERT training by proposing a method that first shares weights across repeated transformer encoder modules to learn common components, then unshares them for fine-tuning, resulting in better model performance and significantly reduced training iterations.
We propose a simple and efficient approach for training the BERT model. Our approach exploits the special structure of BERT that contains a stack of repeated modules (i.e., transformer encoders). Our proposed approach first trains BERT with the weights shared across all the repeated modules till some point. This is for learning the commonly shared component of weights across all repeated layers. We then stop weight sharing and continue training until convergence. We present theoretic insights for training by sharing weights then unsharing with analysis for simplified models. Empirical experiments on the BERT model show that our method yields better performance of trained models, and significantly reduces the number of training iterations.