Preregistering NLP Research
It addresses the need for improved research transparency and reproducibility in NLP, which is incremental as it adapts practices from other fields.
This paper tackles the problem of low adoption of preregistration in NLP research by exploring how to implement it and proposing registered reports to support more rigorous science, aiming to foster community discussion for developing a standard preregistration form.
Preregistration refers to the practice of specifying what you are going to do, and what you expect to find in your study, before carrying out the study. This practice is increasingly common in medicine and psychology, but is rarely discussed in NLP. This paper discusses preregistration in more detail, explores how NLP researchers could preregister their work, and presents several preregistration questions for different kinds of studies. Finally, we argue in favour of registered reports, which could provide firmer grounds for slow science in NLP research. The goal of this paper is to elicit a discussion in the NLP community, which we hope to synthesise into a general NLP preregistration form in future research.