On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research
This addresses the problem of scientific debt in NLP research, calling for more rigor to improve reproducibility and understanding of model advancements, though it is incremental in advocating for methodological changes rather than introducing new techniques.
The paper critiques current language model pre-training research for lacking rigorous ablation studies, which obscures the contributions of different factors to model improvements. It demonstrates that under comparable tuning conditions, earlier baselines like ELMo and GPT-1 can achieve competitive or better performance than BERT, revealing new insights into progress drivers.
This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: "How exactly does each factor contribute to the progress that we have today?" We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how -- under comparable conditions where the baselines are tuned to a similar extent -- these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights. We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.