The Diminishing Returns of Masked Language Models to Science
This work addresses the problem of inefficient resource allocation for researchers and practitioners in scientific domains, showing that scaling strategies may be incremental or ineffective for these tasks.
The study investigated whether scaling up model size, training data, and compute time improves performance on scientific tasks, finding that such increases often do not lead to significant gains (>1% F1) in scientific information extraction.
Transformer-based masked language models such as BERT, trained on general corpora, have shown impressive performance on downstream tasks. It has also been demonstrated that the downstream task performance of such models can be improved by pretraining larger models for longer on more data. In this work, we empirically evaluate the extent to which these results extend to tasks in science. We use 14 domain-specific transformer-based models (including ScholarBERT, a new 770M-parameter science-focused masked language model pretrained on up to 225B tokens) to evaluate the impact of training data, model size, pretraining and finetuning time on 12 downstream scientific tasks. Interestingly, we find that increasing model sizes, training data, or compute time does not always lead to significant improvements (i.e., >1% F1), if at all, in scientific information extraction tasks and offered possible explanations for the surprising performance differences.