METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
This work addresses the high computational cost of pretraining large language models for NLP tasks, offering a more efficient approach.
The authors tackled the problem of efficiently pretraining large autoencoding language models by proposing METRO, a method using model-generated signals, which resulted in METRO-LM models achieving state-of-the-art on GLUE, SuperGLUE, and SQuAD benchmarks with up to 5.4 billion parameters while reducing model size and pretraining cost.
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.