To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks
This work addresses the efficiency of pretraining for resource-rich tasks, suggesting diminishing returns, which is incremental as it builds on existing pretraining methods.
The paper investigates the benefits of pretraining for NLP models as downstream training data increases, finding that the accuracy gap between finetuning BERT-based models and training LSTMs from scratch narrows to within 1% when millions of examples are available.
Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.